- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Psy-MARL, a multi-agent framework trained with SFT followed by RL to generate the next physician response in psychiatric interviews. It separates the task into three roles: patient-state assessment, communication planning, and response generation. The training method fits Bradley–Terry utilities from LLM preferences over policy coalitions and uses Shapley values to assign role-specific rewards for GRPO updates. Experiments on adolescent psychiatric consultation records report improvements over single-agent and multi-agent baselines under LLM-based evaluation, with additional ablations of utility estimation and credit assignment.
Critical strengths
S1. The study uses real psychiatric outpatient dialogues and splits the data by patient before constructing turn-level examples. This helps reduce information leakage from the same patient appearing in both training and test sets.
S2. The two ablations retain the same role structure, SFT initialization, and GRPO setup while examining shared rewards and direct scoring. These comparisons provide targeted evidence for the reward allocation and utility estimation design.
Critical weaknesses
W1: The methodological novelty and supporting insight are limited. BT, Shapley values, and GRPO are established methods, so the main contribution is their integration with workflow-preserving policy coalitions. The convexity and efficiency properties support this construction, but do not establish unbiased policy gradients or better optimization. Such guarantees are not necessary for an empirical contribution, but the current ablations provide limited explanation of why this combination is preferable to simpler alternatives or when its additional complexity is justified.
W2: The evaluation dimensions are underdeveloped and lack reported clinical validation. HDE, MEO, and SSA are motivated by established interviewing principles, but lack detailed scoring anchors for symptom exploration, appropriate open-ended inquiry, and progress toward diagnostic clarification. Interview-stage identification and appropriate transitions are also insufficiently specified. Without clinician validation or illustrative scoring examples, it is difficult to interpret the scores as reliable measures of interviewing quality.
W3: The practical contribution to psychiatric interviewing remains unclear. Evaluation is limited to generating the next response from recorded dialogue histories, using lexical-overlap metrics and a single LLM judge. It does not observe subsequent patient reactions or establish improved information gathering and interview progression. The absence of patient-clustered confidence intervals and training-seed variability further limits confidence in the reported gains. Consequently, the results provide limited insight into which interviewing abilities reliably improve.
Questions for authors
-
Are results available for a single-agent RL baseline with comparable training and inference budgets, particularly one that sequentially performs patient assessment, strategy planning, and response generation?
-
Beyond model judgments of final responses, is there expert evaluation or subsequent-interaction evidence supporting the effectiveness of the communication strategies?
-
Does the clinical improvement reflect better interviewing or simply the LLM judge’s preferences?
4 comments
Response to Reviewer 1 (2/4): W2
W2. Evaluation dimensions: scoring anchors and clinician validation.
HDE, MEO, and SSA are task-specific research rubrics informed by Shea (2016) and Motivational Interviewing, refined with practicing psychiatrists. They are not validated clinical scales, and Appendix A.6 now says so. Thank you for pointing out that anchors and validation were missing. We added the following:
- Five-level behavioral anchors for each dimension and for reference alignment (Appendix A.6, "Scoring anchors and scale correspondence" and "Reference-alignment anchors"; Table 6). They were already part of the original LLM evaluation prompts (0-100 scores in five bands), and we now describe them in detail. For the clinician ratings, clinicians and the LLM re-rating use the same anchors on a 1-5 scale, which makes rating easier for the clinicians.
- What each dimension assesses and does not assess (Appendix A.6). HDE assesses whether a response targets a clinically relevant information gap grounded in the dialogue, and does not assess diagnostic correctness or reward medical terminology, question count, or length. MEO assesses whether the wording supports patient expression and avoids leading, pressuring, or invalidating language, and is not a factual-accuracy criterion. SSA assesses whether continuing or changing the current conversational task is appropriate, and does not reward following a fixed sequence. The three dimensions are scored independently.
- SSA task interpretation (Appendix A.6). Five study-specific task labels (establishing or repairing cooperation; assessment and information gathering; clarification and integration; discussing next steps; summarizing and closing). They do not prescribe a sequence, and a shift of task can be appropriate. Evaluators first select the primary task and then judge task fit without the recorded physician reply or later patient turns.
- Boundary cases and missing-value rules (Appendix A.6 "Boundary cases"; A.8 "Missing ratings"). For example, distrust expressed during assessment can justify a shift toward repairing cooperation. NA means no reasonable opportunity to assess a dimension, and U means insufficient evidence. Poor performance is never NA, and uncertainty between adjacent levels is not U.
- Scoring examples (Appendix A.8, Table 11). For each clinical dimension, we show one context with a higher-rated and a lower-rated candidate response and the rationale, taken from one clinician's ratings, to illustrate which interviewing behaviors each dimension rewards.
- Clinician validation (Section 4.2, Table 3; Appendix A.8, Table 12). Two psychiatrists (doctoral degree, associate chief physician) rated 240 responses blind to model identity and LLM scores, with clinical ratings locked before they saw the recorded physician reply. Quadratic-weighted kappa is 0.622-0.719 between clinicians and 0.636-0.738 between clinicians and the LLM (CIs in Table 12).
Together, these additions specify what the clinical dimensions measure and how they are scored. The clinicians also place Psy-MARL first on all four dimensions, in line with the LLM-based evaluation.
What the ratings show about each dimension. The dimensions capture different interviewing behaviors, and the clinician ratings separate them. Over the strongest baseline, Single-Role-RL, Psy-MARL's clinician mean is higher by +0.27 on HDE (targeting clinically relevant information gaps), +0.49 on MEO (supporting patient expression with open, non-leading wording), and +0.68 on SSA (fit with the current interview stage and appropriate transitions). The largest gains are thus in how a response supports expression and fits the current task, consistent with the design in which the Belief and Strategy agents explicitly assess the patient's state and plan the communication before the response is written. Reference alignment is also the lowest mean for both Psy-MARL (2.808) and Single-Role-RL (2.583), which indicates that the clinical dimensions reward appropriateness beyond closeness to the one recorded reply.
Response to Reviewer 1 (3/4): W3
W3. Practical contribution; offline evaluation; lexical metrics and a single judge; patient-clustered CIs and seeds.
Scope. We study offline next-response generation from recorded dialogue histories. We do not claim improved information gathering, interview progression, or clinical efficacy. The Abstract, Introduction, and Conclusion now state this. The new Limitations and future work paragraph (Section 5) says that the evidence comes from one Mandarin adolescent psychiatric setting, that HDE/MEO/SSA are research rubrics, and that interactive evaluation of patient reactions is future work.
Practical contribution. Psychiatric interviewing requires systematic exploration of symptoms while keeping a collaborative relationship with the patient, and the next question depends on interdependent judgments about the patient's state and how to communicate, grounded in the dialogue context. Predefined procedures or imitation of demonstrated responses adapt poorly to variation in patients' states and expressions. Psy-MARL splits this reasoning into patient-state assessment (Belief), communication planning (Strategy), and response generation, and trains the three roles jointly with role-specific rewards derived from preference judgments on final responses. In this round we added single-policy RL baselines, two non-RL multi-agent baselines, a second backbone, and clinician ratings. Psy-MARL keeps positive margins in all of these comparisons, which indicates that the benefit comes from the role decomposition together with role-specific credit, and not from RL fine-tuning alone. We regard this as a step toward LLM support for psychiatric interviewing in the offline next-response setting.
Metrics and judges. BLEU-2 and ROUGE-L are auxiliary. SFT baselines score higher on them but lose clinically, and the paper discusses this (Section 4.2). The training judge (Qwen3.5-27B) and the evaluation judge (GPT-5.6 Luna) are different models, and each pair is judged in both A/B orders. Clinician ratings cover the clinical dimensions and reference alignment.
Patient-clustered CIs. Thank you for this valuable suggestion. We resample the 40 test patient groups with replacement (2,000 paired replicates, keeping all examples and order-specific judgments of each selected group). All 11 Qwen3-8B comparisons have 95% CIs above zero (Appendix A.6, Table 7), which indicates that the advantages are consistent across patient groups and not driven by a few patients.
Seeds. Because of training resources and time, we did not run independent training seeds. We instead ran an inference-seed sensitivity analysis on Qwen3-8B. With the trained checkpoints fixed, we used five generation seeds (42-46) to regenerate full Belief-Strategy-Response trajectories of Psy-MARL and Single-Role-RL on all 1,830 test contexts. Psy-MARL's net-win margin over Single-Role-RL is 15.16 ± 3.50 pp in reference alignment (range 11.32-19.96) and 9.58 ± 1.88 pp in clinical preference (range 7.50-12.07), mean ± sample SD, and it is positive under every seed, so the advantage does not depend on a particular generation seed. This covers variability at fixed checkpoints only. The results are in Appendix A.6 (Table 8).
Which interviewing abilities improve reliably. Taken together, the new evidence gives a consistent picture within the next-response setting. Clinicians rate the largest gains over Single-Role-RL on MEO (+0.49) and SSA (+0.68), and a smaller gain on HDE (+0.27) (Table 3). The pairwise clinical margins of the LLM judge are positive against every baseline on both backbones, the 11 Qwen3-8B comparisons stay above zero across patient groups, and the margin over Single-Role-RL is positive under all five generation seeds. The most reliable improvements are therefore in supporting patient expression and in choosing the conversational move that suits the current stage, with a smaller gain in targeting information gaps. Whether these translate into better information gathering over a full interview requires interactive evaluation, which we leave to future work.
Response to Reviewer 1 (4/4): Q1-Q3
Q1. Single-agent RL baseline with a comparable budget.
Thank you for the suggestion. We added two single-agent baselines. Single-Role-RL uses one shared policy for the Belief, Strategy, and Response steps, initialized from SFT, with GRPO and BT-fitted full-coalition feedback but no role-specific allocation. Single-Direct-RL generates the response directly, with a 1-5 similarity judge. Psy-MARL's net-win margins (reference alignment / clinical) are:
- over Single-Role-RL: +15.58 / +9.81 pp (Qwen3-8B) and +13.36 / +14.06 pp (Llama-3.1-8B);
- over Single-Direct-RL: +19.33 / +13.74 pp (Qwen3-8B) and +14.82 / +19.52 pp (Llama-3.1-8B).
All methods use the same split, and Single-Role-RL trains for about the same time as Psy-MARL, so the comparison is at comparable cost. Single-Role-RL has the same workflow and BT-fitted feedback as Psy-MARL, differing in the single policy and the absence of role-specific allocation. Psy-MARL's remaining margin therefore reflects multi-role training with role-specific credit. The results are in Tables 1-2 (Section 4.2), with implementation details in Appendix A.5 ("Single-policy RL baselines").
Q2. Expert evaluation or subsequent-interaction evidence.
Expert evaluation is now included (see our reply to W2; Table 3, Section 4.2). Psy-MARL has the highest mean clinician rating on all four dimensions (HDE 3.305, MEO 3.433, SSA 3.567, reference alignment 2.808). The strongest baseline, Single-Role-RL, scores 3.036, 2.942, 2.883, and 2.583. The largest gaps are in MEO (+0.49) and SSA (+0.68), which suggests improvements in how responses support patient expression and fit the current conversational task. The clinicians' ordering of the baselines largely matches the pairwise LLM judge. The exception is SFT MAS, which clinicians rate slightly above Base Single on the clinical dimensions (by at most 0.10), whereas the LLM judge prefers Base Single. This is a small study (2 raters, 60 contexts, Qwen3-8B, 4 methods). We have no evidence on subsequent patient interaction, because recorded dialogues cannot show how patients react to generated responses. We list interactive evaluation as future work.
Q3. Does the gain reflect better interviewing or LLM-judge preference?
We added clinician ratings and a clinician-LLM agreement analysis to make the evidence more reliable. (i) The evaluation judge differs from the training judge, so the results do not simply reproduce the training judge's preference. (ii) Clinicians independently give Psy-MARL the highest mean on the three clinical dimensions. (iii) Clinician-LLM agreement is moderate to substantial (kappa 0.636-0.738). Taken together, these indicate that the gains are not only a matter of LLM-judge preference. The results are in Section 4.2 (Table 3) and Appendix A.8 (Table 12).
Response to Reviewer 1 (1/4): W1
We sincerely thank you for the constructive review. The questions on why the combination works, on the evaluation rubrics and clinician validation, and on single-policy RL baselines led us to add experiments and appendix material that make the paper clearer and the evidence stronger. Our replies come in four comments: W1, W2, W3, and Q1-Q3.
W1. Novelty; why the combination is preferable and when its cost is justified.
We agree that BT, Shapley values, and GRPO are established. Our contribution is the construction that connects them: policy coalitions that keep the full Belief-Strategy-Response workflow (non-members use frozen reference policies), pairwise preferences over adjacent coalitions, joint BT fitting into one utility scale, and exact Shapley allocation that feeds role-specific GRPO groups with fixed prefixes.
Theory. The convexity and efficiency properties apply to a fixed estimated coalition game. They do not imply unbiased policy gradients or better optimization, and we do not claim this. We added this statement to Appendix A.3 ("Properties of Utility Fitting and Credit Allocation"). The effectiveness of the full method is assessed empirically.
Evidence (Section 4.4, Table 5; Tables 1-2). The ablations keep the role structure, SFT initialization, GRPO setup, and reward scaling fixed. Psy-MARL's net-win margins (reference alignment / clinical) are:
Psy-MARL leads every variant that removes one of its components. The credit-quality analysis (R3-W1; Section 4.4, Appendix A.7) adds direct evidence that higher Belief credit predicts better final responses.
Cost (Appendix A.9). Coalition evaluation adds training cost (responses under multiple coalitions and preference judgments), which we reduce with one batched judge request per candidate evaluation and rollout reuse. It occurs only in training, and inference runs the three-agent workflow without coalition evaluation. With the same SFT initialization, batch size 8, and two epochs (2,950 steps), on 8 A800 training GPUs plus a separate two-A800 judge node, Psy-MARL trains in about 59.5 hours (about 595 GPU-hours including the judge node), and Single-Role-RL takes a similar time. Single-Direct-RL takes 31.4 hours (about 314 GPU-hours). The difference comes mainly from the additional sampling under multiple coalitions. Since this cost is paid once during training, and Psy-MARL leads Single-Direct-RL, which trains in about half the time, by +19.33 / +13.74 pp on Qwen3-8B, we consider it a reasonable price for the gain.
Why it works and when it pays off. The evidence points to three sources of the gain.
Our reading is that the added complexity pays off when the final output depends on several interdependent intermediate judgments (here, patient-state assessment and communication planning) and only the final output can be evaluated, because a single outcome score cannot tell the roles apart. The extra cost is incurred only in training.