- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces QH-Bench, a benchmark for evaluating whether LLMs respond safely and helpfully to Chinese adolescents, including risks that only become apparent through conversation history. It contains 715 single-turn questions covering 10 domains and 143 fine-grained scenarios, plus 100 four-turn conversations crossing those domains with 10 dialogue mechanisms, including concealed intent, cross-turn references, and relational pressure. The scenarios are synthetic, generated with LLM assistance and reviewed by researchers. Thirteen open-weight models are evaluated using an automatic judge that assigns scores from −2 (clearly risk-enabling) to +2 (safe and helpful). Main findings: Offline meetings with online contacts are a shared weakness: every evaluated model receives negative scores on more than half of those items. Relational pressure such as invoking loyalty or confidentiality also produces frequent failures. Strong overall performance therefore does not guarantee safety in specific situations.
Critical strengths
- The paper addresses an important, underserved evaluation setting. QH-Bench focuses on Chinese adolescent safety through concrete contexts such as school–guardian communication, youth-platform safeguards, and livestream spending. This contextual specificity makes it a potentially useful resource for identifying safety failures that general-purpose benchmarks may overlook.
Critical weaknesses
-
The reliability of the central safety-failure metric is insufficiently established. Figure S1 shows that 26 of the 60 cases assigned negative scores by the automatic judge receive nonnegative scores from the human annotator. Although one annotator is not definitive ground truth, this substantial disagreement at the threshold defining “risk-enabling” responses raises doubts about the headline failure rates. Independent adjudication, particularly for the highlighted offline-contact and relational-pressure cases, is needed to establish the main empirical conclusions.
-
The multi-turn design provides insufficient scenario replication. Each domain–mechanism combination contains only one dialogue, confounding mechanism difficulty with the particular scenario and wording. Testing 13 models on the same dialogue does not provide independent scenario replication, so the reported mechanism comparisons and cell-level patterns have limited generalizability. This weakens a central contribution: diagnosing systematic safety weaknesses associated with particular conversational mechanisms.
Questions for authors
Figure S1 shows that 26 of the 60 automatically negative cases receive nonnegative human scores. What explains these disagreements, and how are they distributed across domains and mechanisms? Is there independent human validation showing that the headline offline-contact and relational-pressure findings remain valid, including for GLM models excluded from the audit?
Thank you for drawing attention to the disagreements at the negative-score boundary and to the need for scenario replication.
Response to the Question
Q1. What explains the disagreements in Figure S1? Some replies combine a warning with conditional advice. One scorer may read that advice as redirecting the user to a safer option; another may judge that it still helps the user carry out the risky action. Under Table 4, the latter can warrant a negative score despite the warning. This is one plausible source of disagreement; the original ratings did not record item-level reasons. The 26 disagreements span 18 single-turn responses across nine domains and eight trajectories across eight mechanisms. W1 reports the independent check of the two main findings.
Responses to Weaknesses
W1. Independent human validation. Two annotators independently scored all 195 offline-contact responses from the 13 models and 50 relational-pressure (M09) trajectories without seeing the judge’s or each other’s ratings. The original 150-unit audit was not rescored as a set. On offline contact, A and B marked 119/195 and 111/195 responses negative, versus 154/195 for the judge. A found 11 models negative on more than half of the 15 items; B found nine. Nine models met that criterion under both sets of ratings. The audit includes all four GLM variants on offline contact and two on M09. On the ten GLM-4-32B M09 trajectories, each of the three scorers gave six negative scores, with five shared negative cases (Appendix C.4, Tables S4 and S5).
W2. Scenario replication. The 13 outputs per trajectory compare models on the same case. They are not 13 independent scenarios. RQ3 confines mechanism comparisons to the tested trajectories, and RQ4 treats individual cell residuals as diagnostic flags. A recurring interaction would require independently written trajectories in the relevant cells.
Thank you again for the careful review. We would welcome your feedback on whether the independent ratings address your concerns about the two main findings.