acceptodds
Under review as a conference paper at ICLR 2027

VERA-MH: Validation of Ethical and Responsible AI in Mental Health

Abstract

Chatbot usage has increased, including in fields for which they were never developed for–notably mental health support. To that end, we introduce Validations of Ethical and Responsible AI in Mental Health (VERA-MH), a novel clinically-validated evaluation for safety of chatbots in the context of mental health support. The first iteration of VERA-MH focuses on Suicidal Ideation (SI) risks, by assessing how well chatbots can responds to users that might be in crisis. VERA-MH is comprised of three steps: conversation simulation, conversation judging and model rating. First, to simulate conversations with the chatbot under evaluation, another chatbot is tasked with role-playing users based on specific personas. Such user personas have been developed under clinical guidance, to make sure that, among others, multiple risk factors, demographic characteristics and disclosure factors were represented. In the judging step, a second support model is used as an LLM-as-a-Judge, together with a clinically-developed rubric. The rubric is structured as a flow, with a single Yes/No question asked each time, to improve answers' consistency and highlight models' failure modes. In the last stage, results of each conversation are aggregated to present the final evaluation of the chatbot. Together with the framework, we present the result of the evaluations for four leading LLM providers.

Primary area:
datasets and benchmarks
Keywords:
ai evaluation, mental health, chatbot evaluation, chatbots, llm

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M5u4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VERA-MH evaluates API-level chatbot response safety in conversations concerning suicidal ideation (SI). It combines 100 clinically informed personas, adaptive conversations generated by user-role LLMs, and a clinically developed 36-question branching rubric reporting five dimensions separately. The paper reports clinician ratings of simulation realism and human/LLM agreement: Table 3 gives clinician-clinician alpha 0.77 and human-LLM alpha 0.77–0.82, with confidence intervals. It then presents multi-model scorecards from 200 conversations per target. The contribution is an inspectable safety evaluation instrument and empirical profiles, within an explicitly restricted SI, English/US and API-level scope.

Critical strengths
  1. A useful, well-scoped evaluation contribution. The explicit safety/efficacy distinction and SI focus (pp. 1–3, sections 1–3) define a meaningful evaluation problem. Adaptive multi-turn conversations address delayed disclosure and conversation-level failures. This supports acceptance because the design addresses a concrete gap in how chatbot safety is examined.

  2. An inspectable rubric with initial expert evidence and informative profiles. Clinician co-design, item-level questions and routing (sections 4.1–4.2; Tables 6–7, pp. 19–22) make the instrument assessable. Table 3 provides human agreement evidence, and the five-dimensional scorecards expose differences between detection, confirmation and guidance to human care (Figure 1). These features make the output more actionable than a single aggregate score, although actual-pipeline validation still needs clarification.

Critical weaknesses
  1. Validation is not sufficiently assessable and does not directly cover the actual configuration. Section 4.4 and Tables 2–3 (pp. 5–6) do not specify clinician/conversation counts, rater assignment and calibration, the precise agreement estimator and NR treatment. Table 3 evaluates GPT-4o, GPT-5.2, Sonnet 4.5 and Gemini 2.5 Flash, whereas section 5 uses GPT-5.4 as judge. Table 2's user simulators also differ from those used in section 5. Human validation and CIs are present, and version differences do not prove poor transfer. Nevertheless, accessible validation of the actual pipeline, or a justified transfer check, is needed to support trust in the automated measurements.

  2. Scorecard uncertainty and stability are insufficiently characterized. The 200 conversations per target consist of two per persona (section 5, p. 6; section 7.3, p. 8), yet Figures 1–5 pool simulators without reporting per-cell counts, risk-stratified results, CI construction and inferential target, repeated-persona treatment, or bounded stability checks. CIs are reported, including some 0.00 [0.00, 0.00] high-harm cells; without the interval method these cannot establish negligible underlying risk.

Questions for authors
  1. Material clarification and bounded validation: Which anonymous supplement contains the full validation protocol, sample/rater counts, rating units, estimator/NR handling and development/held-out relationship? Does it include expert evidence for the actual GPT-5.4 judge and section 5 user simulators, or a justified transfer check, with selected in-scope gate/high-harm errors?

  2. Material clarification and bounded robustness evidence: Can you provide cell counts/denominators, the CI target and algorithm, repeated-persona treatment, and existing-run results by simulator and risk level? Do the main profiles persist in a limited repeat-run or already-validated-judge sensitivity check?

1 comment

Dear Reviewer M5u4,

Thank you for your thoughtful comments and review of the paper. Thank you for highlighting a miss in your question 2, that we’ll adress below.

On validation.

Your analysis highlighted an actual gap how results were presented. It has now been corrected on the paper, but the TL;DR is that there have been two of human validation studies performed. The first one, limited to 10 personas, didn’t involve got 5.4. The second one, smaller, used GPT5.4 and the results are now reported in Table 4.

Furthermore, a new appendix includes clinical ratings stratified by persona risk level.

Clarification on results

Raw Counts

We have added a new plot in the appendix with the counts. Agreed that currently those number are hard to reconstructs from the published table, as one would have to subtract form the total (200) the portion of each non-relevant ones, and then compute those totals from the percentages. It makes it a little harder to read the table, and we only added for the flagship models. The procedure can be reproduced for other models if desired

CI


CI target and algorithm have been specified as the (conversation, safety dimension) with Krippendorff’s alpha.

Resampling


Resampling was done at the conversation level, and it’s possible that each persona was related, but otherwise they received no other special treatment

Plot by risk level

We have added a new section in the appendix with the results stratified by risk levels. It has not been divided by simulator for space constrains, but the procedure can be done if desired

Repeat run question

I am sorry but I am not sure if I understand the last question “Do the main profiles persist in a limited repeat-run or already-validated-judge sensitivity check?” What is the repeat-run in this context?

Reviewer_AvzGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces VERA-MH, an automated framework for evaluating chatbot safety in conversations involving suicidal ideation. It combines simulated conversations based on 100 clinically informed personas, an LLM judge applying a branching rubric of 36 questions, and model profiles across five dimensions covering risk detection, risk confirmation, referral to human support, supportive communication, and AI boundaries. The contribution is primarily an evaluation framework and clinically informed rubric, accompanied by an evaluation of multiple model families using 200 conversations per evaluated model.

Validation consists of clinician ratings of conversation realism and agreement between clinicians and automated judges. The paper reports clinician–clinician reliability of 0.77 and clinician–LLM reliability between 0.77 and 0.82. Clinicians rated presentation realism more positively than communication style realism. The model evaluations suggest that recognizing potential risk does not necessarily translate into appropriate follow-up questions or guidance toward human care.

Critical strengths

The evaluation addresses a consequential problem through a concrete, inspectable rubric.

The five dimensions distinguish failures with different practical implications, while the branching questions connect ratings to specific chatbot behaviors. This is more useful for diagnosing shortcomings than a single overall score or an unconstrained judge explanation. The explicit focus on suicidal ideation also gives the framework a manageable scope.

The authors make a meaningful effort to ground automation in clinical judgment.

Clinicians contribute to persona and rubric development, and the paper reports both human agreement and human–LLM agreement rather than assuming an automated judge is reliable. The reported agreement is promising. Together with dynamic conversations and an emphasis on making the rubric accessible beyond developers, this gives the framework potential value as a reusable evaluation resource.

Critical weaknesses

The clinical-validation claim is stronger than the evidence that can be assessed from the submission. Section 4.4 does not report the number of clinicians, the number and selection of rated conversations, the allocation of conversations to raters, or whether the final validation set was separate from the examples used to refine the rubric. These omissions matter because the rubric was iteratively revised to improve agreement. The paper also does not adequately specify how the reported α was calculated, including its unit of analysis, treatment of branching and missing ratings, or confidence-interval procedure. Aggregate agreement does not establish how reliably the judge detects high-potential-for-harm cases.

There is also a mismatch between validation and deployment. Table 3 validates several judges, but not GPT-5.4, which is used for the main experiments. Similarly, Table 2 assesses simulator models different from the GPT-5.2 and Opus 4.5 simulators used in those experiments. The overall communication-style realism score of 2.89/5 further warrants caution about treating these conversations as representative of actual chatbot use. I do not expect evidence of treatment efficacy from a safety benchmark, but I do expect validation of the configurations used to support its principal results.

The rubric’s routing and aggregation leave important ambiguity about what the safety profiles measure. In Appendix C, a negative answer to Question 1 routes to Question 3 and then terminates the evaluation. Consequently, conversations without user statements indicating suicide risk do not receive the later assessments of supportive communication and AI boundaries. This could omit harmful chatbot behavior in conversations where the user has not expressed such risk, including behavior that introduces or encourages it. Question 2 also treats acknowledgment at any point as sufficient to avoid its high-harm flag, leaving unclear how delayed recognition after earlier missed signals is captured.

These routing choices interact with the aggregation. Ratings are normalized over conversations deemed relevant, and those conversations can differ across evaluated models because the interaction itself is dynamically generated. The reported percentages therefore describe performance on potentially different subsets and trajectories. Reporting the “Not Relevant” share is helpful, but does not establish that differences reflect chatbot behavior rather than differences in disclosure, conversation length, simulator behavior, or judge routing. The paper needs an audit of these exclusions and analyses by initial persona risk and simulator to make the model comparisons more convincing.

Questions for authors

How many clinicians and conversations contributed to Tables 2 and 3, how were conversations sampled and assigned, and was the final validation performed on conversations held out from rubric development? Please also specify the reliability estimator, its unit of analysis, treatment of branching, and confidence-interval calculation.

Is there clinician validation for the GPT-5.4 judge and the GPT-5.2/Opus 4.5 simulators used in Section 5? In particular, what are the judge’s error rates for high-potential-for-harm cases, rather than only aggregate agreement?

Does the implementation terminate all remaining dimensions when Questions 1 and 3 indicate no user suicide-risk signal? If so, how are harmful chatbot statements in those conversations captured? Similarly, where does the rubric distinguish timely risk recognition from acknowledgment only after several missed opportunities?

Do the main findings remain consistent when results are separated by simulator and initial persona risk level? Please clarify how refusals and early termination are handled, and whether the reported confidence intervals account for the two conversations sharing each persona.

1 comment

Dear AvzG, thank you for your in-depth comments on the paper.

In particular, thank you for highlighting some of the missing details in the presentation of section 4.4, and for highlighting some of the gaps, that have now been corrected.

To answer the specific questions:

Scope of the human validation study

details have been added to answer all the above in section 4.4. As a quick answer; 6 raters, 90 conversation total and 3 reviews per conversation. Krippndoff’s alpha was used for the IRR on (conversation, safety dimension). CI were calculated by resampling conversations. Branching was not treated differently, as the rating on specific question was analzyzed (more on it later). Conversations were generated after rubric changes were finalised, so there should be no contamination between the two.

GPT5.4 validation

Thanks for flagging this gap. As now noted in Section 4.4, there have been 2 human validation studies, with the second, smaller, one validating 5.4

No risk (questions 1 and 3)

That is correct, if both Q1 and Q3 indicate no suicide risk, all remaining dimensions are marked as non-relevant. However, it’s worth noting that the judge LLM gets the full conversation, before making any determination. If harmful statements related to suicidal ideation are present (even towards the end), those would be captured by the judge. The rubric doesn’t currently distinguish between timely vs delayed recognition, which is something we will take back to the clinical team for consideration as a future improvement.

Risk levels

I have added the analysis stratified by the risk level to the appendix, including τ\verbatim−b\tau\verbatim{-}b measure across risk levels.

Refusals

If this is about the user-LLM refusing to role play, I think only a handful (<5) of examples were present and they were discarded.

Early termination

Early termination is handled according to the rubric. The questions that are a terminal state have an END as their corresponding GOTO column. Those are not treated differently, in the sense that when computing the scoring matrix, the "Non Relevant" columns are the first ones to the deal with.

#CI and same persona The bootstraps happens at conversation level, so in this case we are not taking that into account

Reviewer_gFvHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce VERA-MH, a framework for evaluating the safety of LLM-based chatbots in conversations involving suicidal ideation. The framework combines dynamically generated multi-turn conversations, clinically informed user personas, an LLM judging procedure. The evaluation uses 100 personas covering different suicide risk levels, demographic characteristics, and disclosure patterns.

Critical strengths

Clinically informed evaluation. The involvement of clinicians in developing the personas and rubric assessment. Evaluating dynamically generated multi-turn interactions for suicidal ideation, where risk may appear gradually or indirectly.

Critical weaknesses

The framework combines established techniques, including LLM-based user simulation, persona-driven conversations, and LLM-as-a-Judge evaluation. A missed opportunity is an empirical comparison with existing frameworks MindEval and HealthBench. Unsure what VERA-MH adds beyond these framewroks The framework relies entirely on synthetic interactions generated by proprietary LLMs. Clinicians rate the conversation realism which is modest (overall mean 2.89/5). A systematic analysis is needed how different user simulators, judge models, persona distributions etc affect the results.

Questions for authors

How does VERA-MH compare empirically with existing multi-turn mental health evaluation frameworks such as MindEval? What additional safety failures does it identify that existing approaches miss? How sensitive are the reported model rankings and safety assessments to the choice of user simulator and LLM judge? In Table 3 you report clinician–clinician agreement of 0.77 and clinician–LLM agreement of 0.77–0.82. Can you explain if agreement is equally strong for high-risk cases?

1 comment

Thank you gFvH for reading our paper and for your thoughts.

Comparing with MindEval and HealthBench.

MindEval covers different clinical issues than VERA-MH, that is only scoped for Suicidal Ideation, and MindBench does not cover suicide so we feel that a comparison would not be telling us much.

HealthBench is more general and it doesn't separate specific metal health scoring from the rest. MentalHealthBench, the one specific to mental health, was released 2 days before the submission’s deadline.

Sensitivity analysis

The point on the systemic analysis is well taken, and we added more details. Section 4.4 has been reworked to add important details. However, we don't analysed whether IRR is consistent at each risk level. The only experiment we conducted in that respect is on gpt-4o in which the IRR gets slightly higher with increasing risk levels (0.75 -> 0.88). We have, however, added in the appendix the results stratified per risk level, including computing τ−b\tau-b measure across risks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.