- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
As users turn to LLMs for emotional support, this paper aims to address the gap in improving LLM-based emotion regulation. The authors propose a closed-loop framework including an LLM-based planner called MindLoop, which is trained through back-and-forth conversations with simulated users. The planner is optimized using RL and the authors' proposed policy learning approach to improve users' emotion regulation. They then evaluate the trained system in a real-world setting with human participants, using wearable BCI to capture emotional states and adapt the AI's responses accordingly.
Critical strengths
Indicating real-world implementation of MindLoop (a feasibility study) with 23 participants and using wearable BCI to capture emotional states
Critical weaknesses
-
Experiment 1: qwen is used for everything; simulating users, generating counseling responses, and judging how good those responses are. This could create a bias where the RL policy learns to produce responses that qwen itself prefers. the policy is , therefore, optimized to perform well on the same type of criteria for success.
-
Experiment 2: authors didn't show any comparison with a control group where users could use a regular LLM; Experiment 2 results don't indicate that MindLoop is more effective than an ordinary LLM or LLM without the proposed policy; same with adding EEG/BCI; so combining the two experiments makes the paper overwhelming; the scientific contribution of each should be analyzed more rigorously.
-
Move the related work section to the main text of the paper. It is important for readers to know the position of the paper relative to the literature.
Questions for authors
How do the authors ensure that the policy isn't exploiting biases in the simulated environment and LLM judges (100% success rate in simulation?)? With qwen3-30B as the user simulator, counselor, and LLM judge create a risk of reward hacking does RL policy learn to maximize rewards by learning emotion regulation?
9 comments
Response to Reviewer L5dj [2/6]
A1 (continued): Part 2. Cross-Model Robustness and Real-World Validation
We separately tested whether the learned policy transfers beyond its original Qwen-based environment. Simulator and Executor replacement. We held the Planner fixed and replaced the Qwen3-30B User Simulator or Executor individually with DeepSeek-V4-Flash or GLM-5.1, and jointly with DeepSeek-V4.1-Flash-1017 and GLM-5.2. The joint tests used newer models because the original services were no longer available to us.
| Dataset | User Simulator | Executor | AS ↑ | SR ↑ | AT ↓ | EI ↑ | PR ↑ | CTS-R-12 ↑ |
|---|---|---|---|---|---|---|---|---|
| ESConv | Qwen3-30B | Qwen3-30B | 5.00/0.00 | 1.000/0.000 | 17.25/0.59 | 78.35/28.12 | 0.9846/0.12 | 2.83/0.33 |
| ESConv | DeepSeek-V4-Flash | Qwen3-30B | 5.00/0.00 | 1.000/0.000 | 17.18/0.38 | 63.80/10.90 | 1.000/0.000 | 3.18/0.32 |
| ESConv | GLM-5.1 | Qwen3-30B | 5.00/0.00 | 1.000/0.000 | 17.25/0.44 | 66.05/15.11 | 0.969/0.173 | 3.02/0.35 |
| ESConv | Qwen3-30B | GLM-5.1 | 5.00/0.00 | 1.000/0.000 | 17.17/0.38 | 85.22/20.65 | 1.000/0.000 | 3.37/0.29 |
| ESConv | Qwen3-30B | DeepSeek-V4-Flash | 5.00/0.00 | 1.000/0.000 | 17.23/0.47 | 87.24/18.13 | 0.992/0.088 | 3.16/0.35 |
| ESConv | DeepSeek-V4.1-Flash | GLM-5.2 | 5.00/0.00 | 1.000/0.000 | 17.38/0.63 | 42.59/7.63 | 1.000/0.000 | 3.68/0.28 |
| ESConv | GLM-5.2 | DeepSeek-V4.1-Flash | 5.00/0.00 | 1.000/0.000 | 17.33/0.49 | 60.52/16.15 | 0.908/0.291 | 3.96/0.32 |
| ExTES | Qwen3-30B | Qwen3-30B | 5.00/0.00 | 1.000/0.000 | 17.17/0.39 | 92.22/12.75 | 1.000/0.000 | 2.91/0.28 |
| ExTES | DeepSeek-V4-Flash | Qwen3-30B | 5.00/0.00 | 1.000/0.000 | 17.23/0.43 | 63.63/9.88 | 1.000/0.000 | 3.27/0.26 |
| ExTES | GLM-5.1 | Qwen3-30B | 5.00/0.00 | 1.000/0.000 | 17.39/0.49 | 67.85/10.57 | 1.000/0.000 | 3.22/0.28 |
| ExTES | Qwen3-30B | GLM-5.1 | 5.00/0.00 | 1.000/0.000 | 17.20/0.45 | 91.64/14.59 | 0.995/0.071 | 3.44/0.28 |
| ExTES | Qwen3-30B | DeepSeek-V4-Flash | 5.00/0.00 | 1.000/0.000 | 17.23/0.45 | 95.33/9.05 | 1.000/0.000 | 3.30/0.30 |
| ExTES | DeepSeek-V4.1-Flash | GLM-5.2 | 5.00/0.00 | 1.000/0.000 | 17.22/0.51 | 69.10/8.41 | 1.000/0.000 | 3.73/0.26 |
| ExTES | GLM-5.2 | DeepSeek-V4.1-Flash | 5.00/0.00 | 1.000/0.000 | 17.25/0.59 | 73.15/11.03 | 0.990/0.100 | 4.01/0.29 |
All configurations maintained 100% SR and positive mean simulated EI, despite replacing the Qwen-based User Simulator and/or Executor without retraining the Planner. With the Qwen Simulator fixed, replacing the Executor improved CTS-R in all four comparisons and EI in three. Joint replacement also yielded higher CTS-R scores, although EI decreased relative to the original Qwen-based environment. These results suggest that the learned Planner is not exclusively dependent on the original Qwen model family to produce coherent CBT interventions. The variation in EI across simulators further highlights the importance of evaluating intervention quality independently of simulator-generated affective outcomes.
Training Judge replacement. We retrained the Planner with DeepSeek-V4.1-Flash-1017 as the training Judge and evaluated it on the same test sets.
| Dataset | Training Judge | AS ↑ | SR ↑ | AT ↓ | EI ↑ | PR ↑ | CTS-R-12 ↑ |
|---|---|---|---|---|---|---|---|
| ESConv | Qwen3-30B | 5.00/0.00 | 1.000/0.000 | 17.25/0.59 | 78.35/28.12 | 0.9846/0.12 | 2.83/0.33 |
| ESConv | DeepSeek-V4.1-Flash | 5.00/0.00 | 1.000/0.000 | 17.71/1.93 | 79.05/19.50 | 1.000/0.000 | 2.78/0.29 |
| ExTES | Qwen3-30B | 5.00/0.00 | 1.000/0.000 | 17.17/0.39 | 92.22/12.75 | 1.000/0.000 | 2.91/0.28 |
| ExTES | DeepSeek-V4.1-Flash | 5.00/0.00 | 1.000/0.000 | 17.44/2.01 | 90.59/12.43 | 1.000/0.000 | 2.95/0.24 |
Both policies achieved 100% SR on ESConv and ExTES, with comparable EI and independently evaluated CTS-R scores. Notably, replacing the Qwen3-30B training Judge with DeepSeek-V4.1-Flash yielded similar CTS-R scores on ESConv (2.78 vs. 2.83) and ExTES (2.95 vs. 2.91). These results directly address the concern that the learned policy may exploit the scoring preferences of the original Qwen Judge, showing that comparable CBT intervention quality can be obtained when process rewards are provided by a different model family.
Real-world validation. We deployed the simulation-trained Planner without updating it. In the 18-participant physiological-feedback group, the wearable EEG/fNIRS system supported full stage coverage (AS = 5.00, SR = 1.00). Positive PANAS increased by 2.89 and negative PANAS decreased by 2.78 within sessions (both p < 0.05). An independent CBT expert rated these 18 transcripts 2.43 ± 0.32 (/6). These results support the feasibility of deploying the learned policy with real users.
Response to Reviewer L5dj [3/6]
A1 (continued): Part 3. Policy Design and Training Dynamics Analysis
MindLoop incorporates three design choices that reduce opportunities for reward exploitation:
- Planner–Executor Separation: Only the Planner is optimized during RL, while the Executor remains frozen. The Planner selects actions from a constrained CBT stage–strategy space rather than generating arbitrary counseling responses, limiting direct optimization of Judge-preferred textual patterns.
- Structured Process Rewards: Rather than relying on a single holistic LLM preference score, MindLoop combines stage-completion, strategy-appropriateness, and affective rewards. Each stage has three explicit completion criteria evaluated through binary judgments, while strategy appropriateness is scored on a discrete 0/1 scale.
- Affective Feedback Rewards: Affective rewards are derived from segment-level post-affect observations rather than directly from the training Judge's quality assessments, providing a separate optimization signal.
Policy Checkpoint Analysis. To further examine whether policy optimization improves regulation performance rather than merely increasing training rewards, we evaluate six Planner checkpoints throughout RL training on the ESConv test set.
| Checkpoint | Training Reward ↑ | SR (%) ↑ | AT ↓ | EI ↑ | CTS-R (/6) ↑ |
|---|---|---|---|---|---|
| u5 | 8.77 / 6.16 | 59.23/49.33 | 13.56/1.32 | 34.20/25.40 | 2.58 / 0.30 |
| u10 | 9.16 / 6.88 | 72.31/44.92 | 14.02/1.77 | 40.62/32.63 | 2.58 / 0.28 |
| u15 | 11.00 / 6.23 | 83.85/36.95 | 15.37/1.32 | 54.19/35.76 | 2.64/0.30 |
| u20 | 11.66 / 6.55 | 93.85/24.12 | 16.14/1.25 | 59.66/31.94 | 2.69/0.26 |
| u25 | 13.00 / 6.61 | 100.00/0.00 | 17.27/0.83 | 78.18/26.65 | 2.74/0.28 |
| u30 | 13.31 / 6.87 | 100.00/0.00 | 17.25/0.59 | 78.35/28.12 | 2.83 ± 0.33 |
Across training checkpoints, SR increases from 59.23% at u5 to 100% at u25, while EI improves from 34.20 to 78.35 by u30. Since the same stage-transition constraints are applied throughout training, the increase in SR indicates that complete procedural success is not mechanically guaranteed by the constrained action space. The simultaneous improvement in EI further demonstrates that policy optimization improves simulated affective outcomes alongside stage completion.
Importantly, mean training reward increases from 8.77 to 13.31, accompanied by an improvement in independently evaluated CTS-R from 2.58 to 2.83. CTS-R scores are assigned by GPT-6-luna using the expert-designed rubric, independently of the Qwen-based training Judge. Notably, after SR reaches 100% at u25, CTS-R continues to improve from 2.74 to 2.83 at u30, while EI remains relatively stable. This indicates that additional training progress is accompanied by improvements in independently assessed CBT quality even after procedural success has saturated. Together with the cross-model and training-Judge replacement experiments, these results provide converging evidence against the observed improvements being driven solely by exploitation of Qwen-specific reward signals.
We thank the reviewer for these constructive suggestions. We have made the corresponding revisions to the manuscript and included the additional experiments in the Appendix.
We thank the reviewer for these constructive suggestions. We have made the corresponding revisions to the manuscript and included the additional experiments in the Appendix.
Response to Reviewer L5dj [1/6]
We sincerely thank the reviewer for the careful reading and for recognizing the feasibility study using wearable BCI during human interactions. We also appreciate the reviewer’s concerns about Qwen-specific reward bias and the lack of a standard LLM control in the human study, and the suggestion to move Related Work into the main text. We report independent CBT ratings and cross-model and human validation, explain the reward design and checkpoint results, discuss standard LLM baselines in simulation and the human no-feedback comparison, and move Related Work into the main text.
A1: Part 1. Independent CBT Quality Assessment Beyond Training Rewards
Thank you for raising this important concern. A 100% SR indicates procedural completion but does not necessarily reflect effective emotion regulation or CBT intervention quality. To investigate whether the learned policy merely exploits Qwen-specific reward preferences, we introduce independent CBT quality assessments, cross-model evaluations, and training-Judge replacement experiments.
To assess intervention quality without relying on simulator-generated affect labels, we invited a CBT expert to evaluate all 29 real-world dialogues (18 with physiological feedback and 11 without feedback) and 30 sampled simulated dialogues (15 each from ESConv and ExTES), using the original 12-item Cognitive Therapy Scale–Revised (CTS-R). In parallel, GPT-6-luna evaluated the same dialogues using an expert-designed CTS-R rubric, enabling comparison with professional assessments.
Across the 59 dialogues, the LLM ratings showed strong agreement with expert assessments (Spearman ρ = 0.802, Pearson r = 0.813, ICC(A,1) = 0.784, MAE = 0.196). Based on this agreement, we further applied the expert-guided rubric to evaluate CBT intervention quality across the complete simulated test sets, independently of the simulator-generated affect labels.
MindLoop consistently achieves higher CTS-R scores than both GLM-5.1 HP and DeepSeek-V4-Flash HP on ESConv (2.83 vs. 2.63 and 2.06) and ExTES (2.91 vs. 2.63 and 2.22). Importantly, these scores are assigned by GPT-6-luna using an expert-designed rubric, independently of the Qwen-based training Judge and simulator-generated affect labels. The improvements therefore provide complementary evidence that the learned policy achieves better CBT-aligned intervention quality beyond optimizing its original training rewards.