- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VERA-MH evaluates API-level chatbot response safety in conversations concerning suicidal ideation (SI). It combines 100 clinically informed personas, adaptive conversations generated by user-role LLMs, and a clinically developed 36-question branching rubric reporting five dimensions separately. The paper reports clinician ratings of simulation realism and human/LLM agreement: Table 3 gives clinician-clinician alpha 0.77 and human-LLM alpha 0.77–0.82, with confidence intervals. It then presents multi-model scorecards from 200 conversations per target. The contribution is an inspectable safety evaluation instrument and empirical profiles, within an explicitly restricted SI, English/US and API-level scope.
Critical strengths
-
A useful, well-scoped evaluation contribution. The explicit safety/efficacy distinction and SI focus (pp. 1–3, sections 1–3) define a meaningful evaluation problem. Adaptive multi-turn conversations address delayed disclosure and conversation-level failures. This supports acceptance because the design addresses a concrete gap in how chatbot safety is examined.
-
An inspectable rubric with initial expert evidence and informative profiles. Clinician co-design, item-level questions and routing (sections 4.1–4.2; Tables 6–7, pp. 19–22) make the instrument assessable. Table 3 provides human agreement evidence, and the five-dimensional scorecards expose differences between detection, confirmation and guidance to human care (Figure 1). These features make the output more actionable than a single aggregate score, although actual-pipeline validation still needs clarification.
Critical weaknesses
-
Validation is not sufficiently assessable and does not directly cover the actual configuration. Section 4.4 and Tables 2–3 (pp. 5–6) do not specify clinician/conversation counts, rater assignment and calibration, the precise agreement estimator and NR treatment. Table 3 evaluates GPT-4o, GPT-5.2, Sonnet 4.5 and Gemini 2.5 Flash, whereas section 5 uses GPT-5.4 as judge. Table 2's user simulators also differ from those used in section 5. Human validation and CIs are present, and version differences do not prove poor transfer. Nevertheless, accessible validation of the actual pipeline, or a justified transfer check, is needed to support trust in the automated measurements.
-
Scorecard uncertainty and stability are insufficiently characterized. The 200 conversations per target consist of two per persona (section 5, p. 6; section 7.3, p. 8), yet Figures 1–5 pool simulators without reporting per-cell counts, risk-stratified results, CI construction and inferential target, repeated-persona treatment, or bounded stability checks. CIs are reported, including some 0.00 [0.00, 0.00] high-harm cells; without the interval method these cannot establish negligible underlying risk.
Questions for authors
-
Material clarification and bounded validation: Which anonymous supplement contains the full validation protocol, sample/rater counts, rating units, estimator/NR handling and development/held-out relationship? Does it include expert evidence for the actual GPT-5.4 judge and section 5 user simulators, or a justified transfer check, with selected in-scope gate/high-harm errors?
-
Material clarification and bounded robustness evidence: Can you provide cell counts/denominators, the CI target and algorithm, repeated-persona treatment, and existing-run results by simulator and risk level? Do the main profiles persist in a limited repeat-run or already-validated-judge sensitivity check?
Dear Reviewer M5u4,
Thank you for your thoughtful comments and review of the paper. Thank you for highlighting a miss in your question 2, that we’ll adress below.
On validation.
Your analysis highlighted an actual gap how results were presented. It has now been corrected on the paper, but the TL;DR is that there have been two of human validation studies performed. The first one, limited to 10 personas, didn’t involve got 5.4. The second one, smaller, used GPT5.4 and the results are now reported in Table 4.
Furthermore, a new appendix includes clinical ratings stratified by persona risk level.
Clarification on results
Raw Counts
We have added a new plot in the appendix with the counts. Agreed that currently those number are hard to reconstructs from the published table, as one would have to subtract form the total (200) the portion of each non-relevant ones, and then compute those totals from the percentages. It makes it a little harder to read the table, and we only added for the flagship models. The procedure can be reproduced for other models if desired
CI
CI target and algorithm have been specified as the (conversation, safety dimension) with Krippendorff’s alpha.
Resampling
Resampling was done at the conversation level, and it’s possible that each persona was related, but otherwise they received no other special treatment
Plot by risk level
We have added a new section in the appendix with the results stratified by risk levels. It has not been divided by simulator for space constrains, but the procedure can be done if desired
Repeat run question
I am sorry but I am not sure if I understand the last question “Do the main profiles persist in a limited repeat-run or already-validated-judge sensitivity check?” What is the repeat-run in this context?