acceptodds
Under review as a conference paper at ICLR 2027

Psy-MARL: Psychology-Guided Multi-Agent Reinforcement Learning For Psychiatric Interviewing

Abstract

Psychiatric interviewing plays a central role in diagnosis and treatment, requiring clinicians to explore symptoms while attending to the patient's psychological state and communication needs. Advances in large language models offer new opportunities for psychiatric interviewing, yet clinical dialogue reasoning in this highly context-dependent setting remains challenging. To this end, we propose Psy-MARL, a psychology-guided multi-agent reinforcement learning framework inspired by Theory of Mind and Motivational Interviewing. Psy-MARL comprises three agents: the Belief agent assesses the patient’s state, the Strategy agent plans a communication strategy, and the Response agent generates a physician response. Their policies are jointly optimized through reinforcement learning. To obtain informative reward signals and distinguish the contributions of agents, we introduce preference-based coalition credit assignment. This method models role collaboration as a cooperative game, fits coalition utilities from LLM preferences using a Bradley-Terry model, and derives role-specific rewards through Shapley allocation. We train and evaluate Psy-MARL on real-world psychiatric consultation dialogues, using LLM-based pairwise evaluations to assess alignment with recorded physician responses and preference under clinical evaluation criteria. Psy-MARL achieves positive net-win margins against all evaluated single-agent and multi-agent baselines under both protocols, including clinical margins of 6.36-27.19 percentage points against three multi-agent reinforcement learning methods. Ablation studies support the benefits of preference-based coalition utility estimation and role-specific credit allocation. These findings support psychology-guided agent collaboration and preference-based joint training as a promising approach to strengthening clinical reasoning in psychiatric interviewing.

Primary area:
reinforcement learning
Keywords:
Psychiatric Interviewing, Dialogue System, Multi-Agent Reinforcement Learning, Credit-assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WGEZOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Psy-MARL, a multi-agent framework trained with SFT followed by RL to generate the next physician response in psychiatric interviews. It separates the task into three roles: patient-state assessment, communication planning, and response generation. The training method fits Bradley–Terry utilities from LLM preferences over policy coalitions and uses Shapley values to assign role-specific rewards for GRPO updates. Experiments on adolescent psychiatric consultation records report improvements over single-agent and multi-agent baselines under LLM-based evaluation, with additional ablations of utility estimation and credit assignment.

Critical strengths

S1. The study uses real psychiatric outpatient dialogues and splits the data by patient before constructing turn-level examples. This helps reduce information leakage from the same patient appearing in both training and test sets.

S2. The two ablations retain the same role structure, SFT initialization, and GRPO setup while examining shared rewards and direct scoring. These comparisons provide targeted evidence for the reward allocation and utility estimation design.

Critical weaknesses

W1: The methodological novelty and supporting insight are limited. BT, Shapley values, and GRPO are established methods, so the main contribution is their integration with workflow-preserving policy coalitions. The convexity and efficiency properties support this construction, but do not establish unbiased policy gradients or better optimization. Such guarantees are not necessary for an empirical contribution, but the current ablations provide limited explanation of why this combination is preferable to simpler alternatives or when its additional complexity is justified.

W2: The evaluation dimensions are underdeveloped and lack reported clinical validation. HDE, MEO, and SSA are motivated by established interviewing principles, but lack detailed scoring anchors for symptom exploration, appropriate open-ended inquiry, and progress toward diagnostic clarification. Interview-stage identification and appropriate transitions are also insufficiently specified. Without clinician validation or illustrative scoring examples, it is difficult to interpret the scores as reliable measures of interviewing quality.

W3: The practical contribution to psychiatric interviewing remains unclear. Evaluation is limited to generating the next response from recorded dialogue histories, using lexical-overlap metrics and a single LLM judge. It does not observe subsequent patient reactions or establish improved information gathering and interview progression. The absence of patient-clustered confidence intervals and training-seed variability further limits confidence in the reported gains. Consequently, the results provide limited insight into which interviewing abilities reliably improve.

Questions for authors
  1. Are results available for a single-agent RL baseline with comparable training and inference budgets, particularly one that sequentially performs patient assessment, strategy planning, and response generation?

  2. Beyond model judgments of final responses, is there expert evaluation or subsequent-interaction evidence supporting the effectiveness of the communication strategies?

  3. Does the clinical improvement reflect better interviewing or simply the LLM judge’s preferences?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1 (1/4): W1

We sincerely thank you for the constructive review. The questions on why the combination works, on the evaluation rubrics and clinician validation, and on single-policy RL baselines led us to add experiments and appendix material that make the paper clearer and the evidence stronger. Our replies come in four comments: W1, W2, W3, and Q1-Q3.

W1. Novelty; why the combination is preferable and when its cost is justified.

We agree that BT, Shapley values, and GRPO are established. Our contribution is the construction that connects them: policy coalitions that keep the full Belief-Strategy-Response workflow (non-members use frozen reference policies), pairwise preferences over adjacent coalitions, joint BT fitting into one utility scale, and exact Shapley allocation that feeds role-specific GRPO groups with fixed prefixes.

Theory. The convexity and efficiency properties apply to a fixed estimated coalition game. They do not imply unbiased policy gradients or better optimization, and we do not claim this. We added this statement to Appendix A.3 ("Properties of Utility Fitting and Credit Allocation"). The effectiveness of the full method is assessed empirically.

Evidence (Section 4.4, Table 5; Tables 1-2). The ablations keep the role structure, SFT initialization, GRPO setup, and reward scaling fixed. Psy-MARL's net-win margins (reference alignment / clinical) are:

  • over Shared-Reward (role-specific credit removed): +9.66 / +12.40 pp;
  • over Direct-Score (pairwise BT removed, Shapley kept): +23.68 / +15.10 pp;
  • over the new Single-Role-RL, which uses BT-fitted full-coalition feedback without role-specific allocation: +15.58 / +9.81 pp (95% patient-clustered CIs [2.46, 26.15] and [0.28, 27.15]).

Psy-MARL leads every variant that removes one of its components. The credit-quality analysis (R3-W1; Section 4.4, Appendix A.7) adds direct evidence that higher Belief credit predicts better final responses.

Cost (Appendix A.9). Coalition evaluation adds training cost (responses under multiple coalitions and preference judgments), which we reduce with one batched judge request per candidate evaluation and rollout reuse. It occurs only in training, and inference runs the three-agent workflow without coalition evaluation. With the same SFT initialization, batch size 8, and two epochs (2,950 steps), on 8 A800 training GPUs plus a separate two-A800 judge node, Psy-MARL trains in about 59.5 hours (about 595 GPU-hours including the judge node), and Single-Role-RL takes a similar time. Single-Direct-RL takes 31.4 hours (about 314 GPU-hours). The difference comes mainly from the additional sampling under multiple coalitions. Since this cost is paid once during training, and Psy-MARL leads Single-Direct-RL, which trains in about half the time, by +19.33 / +13.74 pp on Qwen3-8B, we consider it a reasonable price for the gain.

Why it works and when it pays off. The evidence points to three sources of the gain.

  1. Role-specific credit. A shared terminal utility mixes the effects of assessment, planning, and response generation. Shapley allocation instead uses utility differences across workflow-preserving substitutions, so each role receives feedback tied to its own marginal contribution. Removing it costs +9.66 / +12.40 pp, and the credit-quality analysis shows that Belief credit predicts better final responses even as collaborators change. In training, the Shapley-based variants also show role-wise reward separation, so the roles do receive different signals (Section 4.4, "Reward dynamics").
  2. Preference-based utility. Independent 1-5 ratings may obscure differences among responses to the same context, whereas pairwise comparisons contrast alternatives directly and BT fitting places them on one utility scale. Replacing them with direct scores costs +23.68 / +15.10 pp.
  3. Neither ingredient is sufficient alone. Single-Role-RL has the full workflow and BT-fitted feedback but no role-specific allocation, and it still trails Psy-MARL by +15.58 / +9.81 pp.

Our reading is that the added complexity pays off when the final output depends on several interdependent intermediate judgments (here, patient-state assessment and communication planning) and only the final output can be evaluated, because a single outcome score cannot tell the roles apart. The extra cost is incurred only in training.

AuthorsOpenReview ↗

Response to Reviewer 1 (2/4): W2

W2. Evaluation dimensions: scoring anchors and clinician validation.

HDE, MEO, and SSA are task-specific research rubrics informed by Shea (2016) and Motivational Interviewing, refined with practicing psychiatrists. They are not validated clinical scales, and Appendix A.6 now says so. Thank you for pointing out that anchors and validation were missing. We added the following:

  • Five-level behavioral anchors for each dimension and for reference alignment (Appendix A.6, "Scoring anchors and scale correspondence" and "Reference-alignment anchors"; Table 6). They were already part of the original LLM evaluation prompts (0-100 scores in five bands), and we now describe them in detail. For the clinician ratings, clinicians and the LLM re-rating use the same anchors on a 1-5 scale, which makes rating easier for the clinicians.
  • What each dimension assesses and does not assess (Appendix A.6). HDE assesses whether a response targets a clinically relevant information gap grounded in the dialogue, and does not assess diagnostic correctness or reward medical terminology, question count, or length. MEO assesses whether the wording supports patient expression and avoids leading, pressuring, or invalidating language, and is not a factual-accuracy criterion. SSA assesses whether continuing or changing the current conversational task is appropriate, and does not reward following a fixed sequence. The three dimensions are scored independently.
  • SSA task interpretation (Appendix A.6). Five study-specific task labels (establishing or repairing cooperation; assessment and information gathering; clarification and integration; discussing next steps; summarizing and closing). They do not prescribe a sequence, and a shift of task can be appropriate. Evaluators first select the primary task and then judge task fit without the recorded physician reply or later patient turns.
  • Boundary cases and missing-value rules (Appendix A.6 "Boundary cases"; A.8 "Missing ratings"). For example, distrust expressed during assessment can justify a shift toward repairing cooperation. NA means no reasonable opportunity to assess a dimension, and U means insufficient evidence. Poor performance is never NA, and uncertainty between adjacent levels is not U.
  • Scoring examples (Appendix A.8, Table 11). For each clinical dimension, we show one context with a higher-rated and a lower-rated candidate response and the rationale, taken from one clinician's ratings, to illustrate which interviewing behaviors each dimension rewards.
  • Clinician validation (Section 4.2, Table 3; Appendix A.8, Table 12). Two psychiatrists (doctoral degree, associate chief physician) rated 240 responses blind to model identity and LLM scores, with clinical ratings locked before they saw the recorded physician reply. Quadratic-weighted kappa is 0.622-0.719 between clinicians and 0.636-0.738 between clinicians and the LLM (CIs in Table 12).

Together, these additions specify what the clinical dimensions measure and how they are scored. The clinicians also place Psy-MARL first on all four dimensions, in line with the LLM-based evaluation.

What the ratings show about each dimension. The dimensions capture different interviewing behaviors, and the clinician ratings separate them. Over the strongest baseline, Single-Role-RL, Psy-MARL's clinician mean is higher by +0.27 on HDE (targeting clinically relevant information gaps), +0.49 on MEO (supporting patient expression with open, non-leading wording), and +0.68 on SSA (fit with the current interview stage and appropriate transitions). The largest gains are thus in how a response supports expression and fits the current task, consistent with the design in which the Belief and Strategy agents explicitly assess the patient's state and plan the communication before the response is written. Reference alignment is also the lowest mean for both Psy-MARL (2.808) and Single-Role-RL (2.583), which indicates that the clinical dimensions reward appropriateness beyond closeness to the one recorded reply.

AuthorsOpenReview ↗

Response to Reviewer 1 (3/4): W3

W3. Practical contribution; offline evaluation; lexical metrics and a single judge; patient-clustered CIs and seeds.

Scope. We study offline next-response generation from recorded dialogue histories. We do not claim improved information gathering, interview progression, or clinical efficacy. The Abstract, Introduction, and Conclusion now state this. The new Limitations and future work paragraph (Section 5) says that the evidence comes from one Mandarin adolescent psychiatric setting, that HDE/MEO/SSA are research rubrics, and that interactive evaluation of patient reactions is future work.

Practical contribution. Psychiatric interviewing requires systematic exploration of symptoms while keeping a collaborative relationship with the patient, and the next question depends on interdependent judgments about the patient's state and how to communicate, grounded in the dialogue context. Predefined procedures or imitation of demonstrated responses adapt poorly to variation in patients' states and expressions. Psy-MARL splits this reasoning into patient-state assessment (Belief), communication planning (Strategy), and response generation, and trains the three roles jointly with role-specific rewards derived from preference judgments on final responses. In this round we added single-policy RL baselines, two non-RL multi-agent baselines, a second backbone, and clinician ratings. Psy-MARL keeps positive margins in all of these comparisons, which indicates that the benefit comes from the role decomposition together with role-specific credit, and not from RL fine-tuning alone. We regard this as a step toward LLM support for psychiatric interviewing in the offline next-response setting.

Metrics and judges. BLEU-2 and ROUGE-L are auxiliary. SFT baselines score higher on them but lose clinically, and the paper discusses this (Section 4.2). The training judge (Qwen3.5-27B) and the evaluation judge (GPT-5.6 Luna) are different models, and each pair is judged in both A/B orders. Clinician ratings cover the clinical dimensions and reference alignment.

Patient-clustered CIs. Thank you for this valuable suggestion. We resample the 40 test patient groups with replacement (2,000 paired replicates, keeping all examples and order-specific judgments of each selected group). All 11 Qwen3-8B comparisons have 95% CIs above zero (Appendix A.6, Table 7), which indicates that the advantages are consistent across patient groups and not driven by a few patients.

Seeds. Because of training resources and time, we did not run independent training seeds. We instead ran an inference-seed sensitivity analysis on Qwen3-8B. With the trained checkpoints fixed, we used five generation seeds (42-46) to regenerate full Belief-Strategy-Response trajectories of Psy-MARL and Single-Role-RL on all 1,830 test contexts. Psy-MARL's net-win margin over Single-Role-RL is 15.16 ± 3.50 pp in reference alignment (range 11.32-19.96) and 9.58 ± 1.88 pp in clinical preference (range 7.50-12.07), mean ± sample SD, and it is positive under every seed, so the advantage does not depend on a particular generation seed. This covers variability at fixed checkpoints only. The results are in Appendix A.6 (Table 8).

Which interviewing abilities improve reliably. Taken together, the new evidence gives a consistent picture within the next-response setting. Clinicians rate the largest gains over Single-Role-RL on MEO (+0.49) and SSA (+0.68), and a smaller gain on HDE (+0.27) (Table 3). The pairwise clinical margins of the LLM judge are positive against every baseline on both backbones, the 11 Qwen3-8B comparisons stay above zero across patient groups, and the margin over Single-Role-RL is positive under all five generation seeds. The most reliable improvements are therefore in supporting patient expression and in choosing the conversational move that suits the current stage, with a smaller gain in targeting information gaps. Whether these translate into better information gathering over a full interview requires interactive evaluation, which we leave to future work.

AuthorsOpenReview ↗

Response to Reviewer 1 (4/4): Q1-Q3

Q1. Single-agent RL baseline with a comparable budget.

Thank you for the suggestion. We added two single-agent baselines. Single-Role-RL uses one shared policy for the Belief, Strategy, and Response steps, initialized from SFT, with GRPO and BT-fitted full-coalition feedback but no role-specific allocation. Single-Direct-RL generates the response directly, with a 1-5 similarity judge. Psy-MARL's net-win margins (reference alignment / clinical) are:

  • over Single-Role-RL: +15.58 / +9.81 pp (Qwen3-8B) and +13.36 / +14.06 pp (Llama-3.1-8B);
  • over Single-Direct-RL: +19.33 / +13.74 pp (Qwen3-8B) and +14.82 / +19.52 pp (Llama-3.1-8B).

All methods use the same split, and Single-Role-RL trains for about the same time as Psy-MARL, so the comparison is at comparable cost. Single-Role-RL has the same workflow and BT-fitted feedback as Psy-MARL, differing in the single policy and the absence of role-specific allocation. Psy-MARL's remaining margin therefore reflects multi-role training with role-specific credit. The results are in Tables 1-2 (Section 4.2), with implementation details in Appendix A.5 ("Single-policy RL baselines").

Q2. Expert evaluation or subsequent-interaction evidence.

Expert evaluation is now included (see our reply to W2; Table 3, Section 4.2). Psy-MARL has the highest mean clinician rating on all four dimensions (HDE 3.305, MEO 3.433, SSA 3.567, reference alignment 2.808). The strongest baseline, Single-Role-RL, scores 3.036, 2.942, 2.883, and 2.583. The largest gaps are in MEO (+0.49) and SSA (+0.68), which suggests improvements in how responses support patient expression and fit the current conversational task. The clinicians' ordering of the baselines largely matches the pairwise LLM judge. The exception is SFT MAS, which clinicians rate slightly above Base Single on the clinical dimensions (by at most 0.10), whereas the LLM judge prefers Base Single. This is a small study (2 raters, 60 contexts, Qwen3-8B, 4 methods). We have no evidence on subsequent patient interaction, because recorded dialogues cannot show how patients react to generated responses. We list interactive evaluation as future work.

Q3. Does the gain reflect better interviewing or LLM-judge preference?

We added clinician ratings and a clinician-LLM agreement analysis to make the evidence more reliable. (i) The evaluation judge differs from the training judge, so the results do not simply reproduce the training judge's preference. (ii) Clinicians independently give Psy-MARL the highest mean on the three clinical dimensions. (iii) Clinician-LLM agreement is moderate to substantial (kappa 0.636-0.738). Taken together, these indicate that the gains are not only a matter of LLM-judge preference. The results are in Section 4.2 (Table 3) and Appendix A.8 (Table 12).

Reviewer_WZ7ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Psy-MARL, a psychology-guided multi-agent reinforcement learning framework inspired by Theory of Mind and Motivational Interviewing with three agents: the Belief agent assesses the patient’s state, the Strategy agent plans a communication strategy, and the Response agent generates a physician response. The emprical study is evaluated on a small-scale, private dateset with 350 real psychiatric consultations.

Critical strengths
  • This work proposed a psychology-guided multi-agent design that is informed by Theory of Mind and Motivational Interviewing, where specialized roles for patient-state assessment and communication planning jointly guide physician response generation.
  • This work proposed preference-based coalition credit assignment. This fits coalition utilities from LLM preference judgments using a Bradley–Terry model and applies Shapley allocation to derive role-specific rewards. This connects feedback on final responses to the joint optimization of intermediate reasoning and communication decisions.
Critical weaknesses
  • The evaluation uses a single, private dataset comprising only 350 psychiatric consultations. This limited scale and lack of access to the data make it difficult to assess the robustness and generalizability of the findings.
  • All baselines use the same backbone. Would the results and conclusions hold with other backbone models? Please clarify how backbone variants affect the main findings. In addition, given the proposed multi-agent design, the comparison with RL-based methods alone seems limited. Why are other multi-agent approaches not included as baselines?
  • The evaluation results for Psy-MARL on the Reference Alignment and Clinical criteria are missing, and the paper does not explain this omission.
Questions for authors
  • Why is the proposed model not evaluated on additional publicly available datasets? If such an evaluation is not feasible, please discuss the model’s limitations and its applicability to broader, more general settings.
  • Why are non-RL multi-agent approaches not included as baselines?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2 (1/2): W1 and Ethics

We sincerely thank you for the review comments. The suggestions on additional backbones, non-RL multi-agent baselines, and the table notation helped us revise the original manuscript and strengthened the experiments, and the attention to the protection of minors helped us state the data and ethics information more completely. Our replies come in two comments: W1 and Ethics here, and W2, W3, Q1, Q2 in the next.

W1. A single private dataset of 350 consultations.

We understand the concern about the data and would like to clarify. The corpus consists of real outpatient psychiatric consultations with patients aged 10-18. It is hard to scale quickly because collection requires clinical access, guardian consent, and minor assent. The 350 consultations, one from each of 350 patients, yield 13,624 turn-level examples (11,794 training, 1,830 test). The split is by patient (310 for training, 40 for testing). The patients cover more than 20 psychiatric categories, including depressive state, anxiety state, emotional disorders, dissociative (conversion) disorders, and somatoform disorders. Each consultation is multi-turn, with about 39 physician-response examples per patient on average. On the test set, the dialogue history has a median of 23 and a mean of 28.2 exchanges (maximum 109).

For scale, published real-consultation corpora in psychiatry are of comparable or smaller size. PDCH (Cao et al., 2025, Scientific Data) has 100 consultations with adult patients with depression, and Howes et al. (2013, Biomedical Informatics Insights) analyze 138 consultations with patients with schizophrenia or schizoaffective disorder from three centers. These populations differ from ours, so we cite them only for typical scale.

Because the data involve minors, access is controlled. The de-identified corpus will be available through controlled access upon acceptance, under a data-use agreement, as stated in the Data Availability section. Unredacted transcripts will not be released. For verification during review, we have provided temporary, review-only access to the de-identified dataset to the reviewers and chairs in a separate comment ("Dataset for review purposes only"). We also state the scope explicitly in the revision (Section 5, Limitations and future work). Our evidence concerns offline next-response generation in one Mandarin adolescent psychiatric setting, and generalization across institutions, age groups, populations, and languages will be the direction of our future work.

Ethics: protection of minors.

We would like to clarify the ethics questions further. The study uses de-identified text for offline training and evaluation, involves no prospective clinical deployment, and no model output was given to patients. The consultations were part of routine outpatient care, and the study added no procedures or contact for the patients, so the experiments did not affect the participating minors. The IRB of the collaborating hospital approved data collection and its use for model training and evaluation, and the approval number has been provided to the Area Chair (withheld here for double-blind review). Guardians gave written informed consent, and minors gave assent. Transcripts were manually reviewed, and personal identifiers were removed or replaced with placeholders before analysis.

AuthorsOpenReview ↗

Response to Reviewer 2 (2/2): W2, W3, Q1, Q2

W2. Same backbone for all baselines; other multi-agent baselines.

Backbone. We added Llama-3.1-8B-Instruct for Psy-MARL and the main single-policy and SFT/Base baselines. Psy-MARL's net-win margins (reference alignment / clinical) are +14.72 / +16.17 pp over Base Single, +17.06 / +28.84 pp over SFT MAS, +13.36 / +14.06 pp over Single-Role-RL, and +14.82 / +19.52 pp over Single-Direct-RL. Psy-MARL also keeps positive margins over the remaining Llama baselines (reference alignment +11.27 pp over Base MAS and +16.71 pp over SFT Single). These results show that the advantage is not specific to Qwen3-8B, which strengthens the evidence that it comes from the method and not from a particular backbone. They are in Tables 1 and 2, panel (b) (Section 4.2), with training settings in Appendix A.4.

Multi-agent baselines. Thank you for the suggestion. Besides MAPoRL, MARFT, and MADA-RL, we added two non-RL multi-agent methods, MultiAgentESC and AutoCBT. On Qwen3-8B, Psy-MARL's margins are +21.25 / +19.10 pp over MultiAgentESC and +19.72 / +20.60 pp over AutoCBT. Psy-MARL therefore leads all five multi-agent methods on both protocols (reference alignment +7.89 to +28.40 pp; clinical +6.36 to +27.19 pp). Both new methods were designed for emotional support or CBT counseling, and we adapted them to this task. The results are in Section 4.3 (Table 4), and the implementation details are in Appendix A.5.

W3. Psy-MARL's results appear to be missing in the tables.

Thank you for pointing out this ambiguity, but the Psy-MARL's results IS NOT missing. In the comparison and ablation tables, Win and Net are pairwise results of Psy-MARL versus the row method, not a score for the row method. For example, against MADA-RL, Psy-MARL has net-win margins of +7.89 pp (reference alignment) and +6.36 pp (clinical). The "--" in Psy-MARL's own row marks a self-comparison and is not applicable. We relabeled the columns "Psy-MARL vs. Method" in Tables 1 and 2, and the captions of Tables 1, 2, 4, and 5 now state the comparison direction and that "--" marks a self-comparison.

Q1. Why no additional public datasets?

Thank you for this suggestion. This study is limited to real adolescent psychiatric consultations, so our evaluation is built on the data we could obtain with ethics approval and guardian consent. We agree that validation on additional datasets would strengthen the work. The new Limitations and future work paragraph (Section 5) states the offline, single-setting scope. We plan to extend the method to other datasets, institutions, and languages as suitable data become available, and to share our corpus through controlled access so that others can build on it.

Q2. Why were non-RL multi-agent approaches not included?

Thank you for the suggestion. We added MultiAgentESC and AutoCBT (see W2; Section 4.3, Table 4). Psy-MARL leads both by +19.10 to +21.25 pp, which further strengthens the evidence that its advantage is not limited to comparisons with RL-based multi-agent methods.

Reviewer_yDrGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains three agents for psychiatric interviewing using BT utilities and Shapley rewards, with evaluation on real consultation records.

Critical strengths

1.The coalition design keeps the full workflow intact when estimating each agent’s contribution. 2.The two controlled ablations help separate the effects of utility estimation and reward allocation. 3.The data come from real consultations, and splitting by patient reduces the risk of leakage.

Critical weaknesses

1.It remains unclear whether candidates with higher credit reliably help the full team as the other agents change. 2.A comparable single-agent RL baseline is missing, so the need for three separately trained agents is not well established. 3.The clinical evaluation relies on one LLM judge without reported validation by clinicians. 4.The paper does not report variation across training seeds or confidence intervals accounting for multiple turns from the same patient.

Questions for authors

1.Have you checked whether candidates with higher credit produce better final responses as the other agents are updated? 2.How does the method compare with a single agent trained with RL to perform the same three stages under a comparable budget? 3.Are any clinician assessments available to check whether the LLM scores reflect better interviewing? 4.Can you report results across training seeds and confidence intervals computed by resampling patients?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (1/2): W1 / Q1

We sincerely thank you for the insightful review. The question of whether credit stays informative as the collaborating agents change led us to a new analysis that gives a direct view of how Psy-MARL works. The comments on single-policy baselines, clinician validation, and patient-level uncertainty also improved the paper. Our replies come in two comments: W1/Q1 here, and W2-W4 in the next.

W1 / Q1. Do higher-credit candidates still help the full team as other agents change?

Credit is not computed on intermediate text in isolation. Each Belief candidate is completed into a final response through the downstream agents. Coalition members use the current role policies and non-members use frozen reference policies, so credit is re-estimated under the current collaborators. We agree that this is an important point for understanding how Psy-MARL works, so we added a direct test (Appendix A.7, "Credit-Quality Consistency under Collaborator Updates", Table 9; summarized in Section 4.4):

  • For 40 held-out contexts (one per patient group), a fixed Belief checkpoint produced four candidates. We recomputed candidate-level Shapley credits with Strategy and Response checkpoints from step 1200 (intermediate) and step 2800 (late).
  • For each context, we took the highest- and lowest-credit candidates and had the downstream agents generate two new continuations, independent of those used to estimate credit. We evaluated them with the standard reference-alignment and clinical protocols in both A/B orders, with the evaluator blind to credit and checkpoint.

Net-win margin of high-credit over low-credit continuations (pp, 95% patient-clustered CI):

Credit ckptCollaborator ckptReference alignmentClinical
IntermediateIntermediate+4.21 [1.82, 12.07]+17.73 [3.05, 29.12]
LateLate+7.93 [4.14, 18.65]+21.17 [3.69, 32.16]
IntermediateLate+5.90 [0.22, 14.23]+15.33 [2.18, 27.43]

All six intervals exclude zero. Higher Belief credit therefore predicts better final responses, both when credit is recomputed for the current collaborators and when an earlier credit ranking is used after collaborators are updated. These Belief-candidate experiments show that higher credit goes with better outcomes for the team's final response. Complementing this, the training-level Shared-Reward ablation (same role structure, SFT initialization, and GRPO setup) gives margins of +9.66 pp (reference alignment) and +12.40 pp (clinical). Replacing role-specific credit with a shared team reward lowers performance, which is consistent with role-specific credit providing useful training signals.

AuthorsOpenReview ↗

Response to Reviewer 3 (2/2): W2 / Q2, W3 / Q3, W4 / Q4

W2 / Q2. Single-agent RL baseline with a comparable budget.

Thank you for the suggestion. We added Single-Role-RL (one shared policy for the Belief, Strategy, and Response steps, with BT-fitted full-coalition feedback and no role-specific allocation) and Single-Direct-RL (direct response generation). Psy-MARL's net-win margins (reference alignment / clinical) are:

  • over Single-Role-RL: +15.58 / +9.81 pp (Qwen3-8B) and +13.36 / +14.06 pp (Llama-3.1-8B);
  • over Single-Direct-RL: +19.33 / +13.74 pp (Qwen3-8B) and +14.82 / +19.52 pp (Llama-3.1-8B).

Psy-MARL thus leads single-policy RL on both backbones and both protocols, under the same SFT initialization and GRPO setup. Single-Role-RL trains for about the same time as Psy-MARL (about 59.5 hours), so the comparison is at comparable cost. Since Single-Role-RL differs from Psy-MARL only in using a single policy without role-specific allocation, the remaining margin reflects the benefit of multi-role training with role-specific credit. These baselines are new in this revision and appear in Section 4.1 (Baselines), Tables 1-2 and the clinical-metrics paragraph of Section 4.2, and Appendix A.5; training cost is in Appendix A.9.

W3 / Q3. Clinician validation of the LLM judge.

Thank you for the suggestion. We added clinician rating evidence (Section 4.2, Table 3; Appendix A.8, Tables 11-12). Two psychiatrists rated 240 responses (60 contexts, 4 methods) blind to model identity, using the same five-level anchors as the LLM re-rating. Quadratic-weighted kappa is 0.622-0.719 between clinicians and 0.636-0.738 between clinicians and the LLM. Psy-MARL has the highest clinician mean on all four dimensions. The clinicians' ordering of the baselines largely matches the LLM judge. The exception is SFT MAS versus Base Single on the clinical dimensions: clinicians rate SFT MAS higher by at most 0.10, whereas the LLM judge prefers Base Single. Overall, clinicians and the LLM judge agree that Psy-MARL ranks first, and their agreement shows that the LLM-based evaluation is reasonably consistent with expert judgment under shared anchors.

W4 / Q4. Training seeds and CIs that account for repeated turns per patient.

CIs. Thank you for the suggestion. We added patient-clustered CIs for all Qwen3-8B comparisons. We resample the 40 test patient groups with replacement (2,000 paired replicates), keeping all examples and both A/B-order judgments of each selected group. All 11 comparisons have 95% CIs above zero (Appendix A.6, Table 7), which indicates that the advantages are consistent across patient groups.

Seeds. Because of computing resources and time, we could not add independent training seeds. We instead ran an inference-seed analysis (five generation seeds, trained checkpoints fixed; Appendix A.6, Table 8). Psy-MARL's net-win margins over Single-Role-RL are 15.16 ± 3.50 pp (reference alignment) and 9.58 ± 1.88 pp (clinical), positive under every seed. This shows that the advantage over this closest single-policy baseline does not depend on a particular generation seed.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

Global Response

We sincerely thank all reviewers for their careful reading and constructive suggestions. The requests for single-policy RL baselines, clinician validation, patient-clustered uncertainty estimates, a second backbone, and clearer scoring criteria led us to new experiments and analyses, and we believe they have made the paper substantially stronger. All revisions are marked in blue in the updated PDF. We summarize the common points here and answer each reviewer in separate comments.

Scope. Our evidence concerns next-response generation from recorded dialogues in one Mandarin adolescent psychiatric setting. We do not claim improved interview progression, patient outcomes, or clinical efficacy. The Abstract, Introduction, and Conclusion now state this, and the Conclusion includes a new Limitations and future work paragraph.

Main additions (section and table numbers refer to the updated PDF).

  1. Single-policy RL baselines (Single-Direct-RL and Single-Role-RL): one runs the Belief-Strategy-Response workflow with a single policy, the other generates the response directly. Psy-MARL leads both on both protocols and both backbones, by +9.81 to +19.52 pp in net-win margin, so its gain is not explained by RL training or the three-step workflow alone (R1-Q1, R3-W2; Tables 1-2, Appendix A.5).
  2. Clinician study: two psychiatrists rated 240 responses (60 contexts, 4 methods) with five-level behavioral anchors. Psy-MARL has the highest clinician mean on all four dimensions, and clinician-LLM agreement is moderate to substantial (kappa 0.636-0.738) (R1-W2/Q2/Q3, R3-W3; Table 3, Appendix A.8).
  3. Patient-clustered 95% bootstrap CIs and inference-seed analysis: all 11 Qwen3-8B comparisons have CIs above zero, and Psy-MARL's margin over Single-Role-RL is positive under all five generation seeds (R1-W3, R3-W4; Appendix A.6). Because of training resources and time, we did not run independent training seeds, and added the inference-seed analysis instead.
  4. A second backbone and non-RL multi-agent baselines: on Llama-3.1-8B-Instruct, Psy-MARL keeps positive margins over all six baselines on both protocols. On Qwen3-8B it leads MultiAgentESC and AutoCBT by +19.10 to +21.25 pp (R2-W2; Tables 1, 2, 4).
  5. Credit-quality consistency analysis: it shows that higher credit assigned to a Belief candidate predicts better final responses, including after the collaborating agents are updated; all six 95% CIs exclude zero (R3-W1; Section 4.4, Appendix A.7).
  6. Scoring anchors, SSA task interpretation, boundary cases, NA/U rules, and scoring examples: we added five-level scoring anchors, the SSA task interpretation, boundary cases, the NA/U rules, and scoring examples with rationales (Appendix A.6, A.8). They help readers understand which interviewing abilities HDE, MEO, and SSA measure, and which of them Psy-MARL improves (R1-W2).
  7. Training cost, scope of the theoretical properties, and table notation: following the reviewers' suggestions, we added the relevant details (training cost in Appendix A.9, the scope of the theory in Appendix A.3, and the notation in the captions of Tables 1, 2, 4, and 5), which improves the clarity and transparency of the paper (R1-W1, R2-W3).

Dataset. The corpus contains consultations with minors, so public release is not possible. For verification during review, we have provided temporary, review-only access to the de-identified dataset to the reviewers and chairs in a separate comment ("Dataset for review purposes only"). We describe the data in our reply to Reviewer 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.