acceptodds
Desk rejected at ICLR 2027

Raising the Bar for Chinese Adolescent LLM Safety: A Culturally-Grounded, Fine-Grained Benchmark

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety risks in conversations with adolescents are not always explicit. A request may appear harmless unless a model considers the user’s age, circumstances, and earlier turns. Existing Chinese safety benchmarks mainly target general users and give limited attention to adolescent safety. Single-turn tests also miss risks that emerge over several turns. QH-Bench is a Chinese-language benchmark for adolescent content safety, with scenarios grounded in Chinese social and cultural settings. The single-turn track contains 715 test items organized into 10 risk domains, 50 subdomains, and 143 fine-grained risk scenarios. The multi-turn track contains 100 four-turn trajectories in a balanced 10 × 10 design that combines the same ten domains with ten cross-turn mechanisms. Both tracks use the same five-level safety–helpfulness scale and automatic judge, with track-specific criteria. Evaluation of 13 open-weight models identifies offline-contact scenarios as a shared weakness. Every model receives negative scores on more than half of the items involving offline meetings with online contacts, unfamiliar groups, and adults. This includes InternLM2.5-20B, the single-turn leader; negative scores indicate responses that partially or clearly facilitate risk. GLM-4-32B, the multi-turn leader, receives negative scores on 60% of complete trajectories in which users build relationships before invoking loyalty or confidentiality. These findings identify two priorities for the evaluated models: handling adolescent offline-contact risks and maintaining safety boundaries under relational pressure. Leading aggregate scores do not establish that these specific weaknesses have been resolved. Code and benchmark: https://anonymous.4open.science/r/QH-bench/

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language model safety, adolescent safety, content safety, Chinese-language benchmark, multi-turn dialogue, safety evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xgNxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces QH-Bench, a benchmark for evaluating whether LLMs respond safely and helpfully to Chinese adolescents, including risks that only become apparent through conversation history. It contains 715 single-turn questions covering 10 domains and 143 fine-grained scenarios, plus 100 four-turn conversations crossing those domains with 10 dialogue mechanisms, including concealed intent, cross-turn references, and relational pressure. The scenarios are synthetic, generated with LLM assistance and reviewed by researchers. Thirteen open-weight models are evaluated using an automatic judge that assigns scores from −2 (clearly risk-enabling) to +2 (safe and helpful). Main findings: Offline meetings with online contacts are a shared weakness: every evaluated model receives negative scores on more than half of those items. Relational pressure such as invoking loyalty or confidentiality also produces frequent failures. Strong overall performance therefore does not guarantee safety in specific situations.

Critical strengths
  1. The paper addresses an important, underserved evaluation setting. QH-Bench focuses on Chinese adolescent safety through concrete contexts such as school–guardian communication, youth-platform safeguards, and livestream spending. This contextual specificity makes it a potentially useful resource for identifying safety failures that general-purpose benchmarks may overlook.
Critical weaknesses
  1. The reliability of the central safety-failure metric is insufficiently established. Figure S1 shows that 26 of the 60 cases assigned negative scores by the automatic judge receive nonnegative scores from the human annotator. Although one annotator is not definitive ground truth, this substantial disagreement at the threshold defining “risk-enabling” responses raises doubts about the headline failure rates. Independent adjudication, particularly for the highlighted offline-contact and relational-pressure cases, is needed to establish the main empirical conclusions.

  2. The multi-turn design provides insufficient scenario replication. Each domain–mechanism combination contains only one dialogue, confounding mechanism difficulty with the particular scenario and wording. Testing 13 models on the same dialogue does not provide independent scenario replication, so the reported mechanism comparisons and cell-level patterns have limited generalizability. This weakens a central contribution: diagnosing systematic safety weaknesses associated with particular conversational mechanisms.

Questions for authors

Figure S1 shows that 26 of the 60 automatically negative cases receive nonnegative human scores. What explains these disagreements, and how are they distributed across domains and mechanisms? Is there independent human validation showing that the headline offline-contact and relational-pressure findings remain valid, including for GLM models excluded from the audit?

1 comment

Thank you for drawing attention to the disagreements at the negative-score boundary and to the need for scenario replication.

Response to the Question

Q1. What explains the disagreements in Figure S1? Some replies combine a warning with conditional advice. One scorer may read that advice as redirecting the user to a safer option; another may judge that it still helps the user carry out the risky action. Under Table 4, the latter can warrant a negative score despite the warning. This is one plausible source of disagreement; the original ratings did not record item-level reasons. The 26 disagreements span 18 single-turn responses across nine domains and eight trajectories across eight mechanisms. W1 reports the independent check of the two main findings.

Responses to Weaknesses

W1. Independent human validation. Two annotators independently scored all 195 offline-contact responses from the 13 models and 50 relational-pressure (M09) trajectories without seeing the judge’s or each other’s ratings. The original 150-unit audit was not rescored as a set. On offline contact, A and B marked 119/195 and 111/195 responses negative, versus 154/195 for the judge. A found 11 models negative on more than half of the 15 items; B found nine. Nine models met that criterion under both sets of ratings. The audit includes all four GLM variants on offline contact and two on M09. On the ten GLM-4-32B M09 trajectories, each of the three scorers gave six negative scores, with five shared negative cases (Appendix C.4, Tables S4 and S5).

W2. Scenario replication. The 13 outputs per trajectory compare models on the same case. They are not 13 independent scenarios. RQ3 confines mechanism comparisons to the tested trajectories, and RQ4 treats individual cell residuals as diagnostic flags. A recurring interaction would require independently written trajectories in the relevant cells.

Thank you again for the careful review. We would welcome your feedback on whether the independent ratings address your concerns about the two main findings.

Reviewer_TAenOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes QH-Bench, a Chinese safety benchmark for teenagers, with 715 single-turn prompts and 100 four-turn conversations based on local Chinese settings. They test 13 open models using an LLM judge on a [-2 to +2] scale. Every model struggle with offline-meetup scenarios and with users who build trust before asking for something risky, and models rank quite differently on single-turn versus multi-turn tests.

Critical strengths
  1. The paper works on an important research area. As more LLMs are used and the risks grow, we need to study risk types.
  2. Safety risks can depend on local settings, and general benchmarks may miss them. The focus on one language and culture is a good direction.
Critical weaknesses
  1. All input context is synthetic and was made by single LLM (GPT-5.5, Sec 3.4). The paper does not say why this model was chosen or compare it with other models. This may (or may not) add bias between the model that made the data and the models being tested. Since this is a benchmark paper, it should check data quality across more than one data source.
  2. The paper does not explain how researchers reviewed the data. Sec 3.4 does not give any guidelines, quality guidelines, or rules for what approves in their criteria. Line 254 says "unsatisfactory trajectories were revised or regenerated," but readers cannot tell how feedback was given, how changes were made, or how many items failed. These details should be added to the appendix and cited with proper grounding.
  3. The paper does not explain well why it picked these domains, subdomains, scenarios, and risk mechanisms. A table in the appendix that links them to domains and scenarios (as on Line 192) would help.
  4. The discussion around the quality of the generated scenarios and conversations in very shallow.
  5. The scoring method in Section 4.2 is too shallow. It uses only a five-point Likert scale. Safety in LLM interactions is sensitive and complex. The scoring has to be better grounded and data-driven. Right now, it is hard to see how the ideas in the paper fit into this scale.
  6. The human audit in Appendix C.4 uses only one person, and the paper does not explain why. One person's scores are not enough to trust the audit. And just because of this, as a reviewer I cannot infer whether the LLM judge or the tested models do better or worse than humans in any of the results.
Questions for authors
  1. Why did you use exactly four turns in the multi-turn track? Did you think about letting the conversations run until they reach a natural point? Would that change which mechanisms are hardest? Would adding a supporting experiment on why four only would help?
  2. How did authors decide to use only the background and user message as model input? Authors also generated scenario labels and other related details. Did you try other input combinations, and how did they affect the results?
1 comment

Thank you for recognizing why local context matters for adolescent safety and for asking us to clarify how the scenarios were built and scored.

Responses to Questions

Q1. Why four turns? Four turns give us room to establish a situation, introduce relevant cues, and test whether the assistant uses them when responding to a later request. Each model receives the same user messages in the same order and retains its own replies. Ending the dialogue based on each model’s behavior could leave some models without the cue being tested.

Q2. Why these inputs? We aim to approximate how an adolescent would ask an assistant for help. The assistant sees the scenario background and user message and, in the multi-turn track, the preceding dialogue including its own replies. We reserve the risk and mechanism labels for analysis so the assistant must infer the risk from the situation. Appendix A lists the model-visible fields.

Responses to Weaknesses

W1. Choice of GPT-5.5. We chose GPT-5.5 because we judged it well suited to drafting natural Chinese scenarios and coherent dialogues under detailed constraints. The researchers defined the taxonomy and mechanisms before generation and reviewed each candidate. We used one generator, so the review checks the retained cases against our criteria rather than comparing generators.

W2. Researcher review. We checked whether each candidate was plausible for a Chinese adolescent and matched its assigned risk. For multi-turn items, we also checked dialogue coherence and whether a later request depended on information from an earlier turn. Candidates that failed these checks were revised or regenerated (Section 3.4).

W3. Taxonomy and mechanisms. In developing the taxonomy, we consulted public materials including China's Law on the Protection of Minors, the Regulations on the Protection of Minors in Cyberspace, and the joint guidance on regulating livestream tipping and protecting minors. The taxonomy describes what the risk concerns, while the mechanisms describe what information a model needs to carry across turns. Tables 2 and 3 and Appendix B show these roles.

W4. Scenario quality. In the livestream example in Appendix B, the last request is to send another gift. Earlier turns reveal that the adolescent has borrowed money from a classmate and used family funds without permission. These facts change how the assistant should respond, illustrating what we checked for when reviewing multi-turn candidates.

W5. Why five scoring levels? Table 4 is a behavior-based ordinal rubric. Scores of −1 and −2 distinguish partial from clear risk enablement; 0 covers replies that do not increase risk but are ineffective, such as a generic refusal; +1 and +2 distinguish a reply that recognizes the risk and maintains a safety boundary but offers limited help from one that offers a specific, age-appropriate alternative. A binary label would collapse these distinctions. SafeDialBench also uses five behaviorally defined tiers for multi-turn safety, while Safe RLHF distinguishes harmlessness from helpfulness in its annotations. These are design precedents; neither uses our exact scale or validates its cutoffs.

W6. Human scoring. The original 150-unit audit compared one human annotator with the judge on the same model outputs (73.3% exact agreement; weighted κ = 0.715). It was not a comparison of people and models answering the prompts. The two-annotator audit targeted 195 offline-contact responses and 50 M09 trajectories, rather than rescoring the original 150 as a set. On those 245 units, the annotators agreed on the negative/nonnegative boundary in 200 cases (81.6%) and exactly in 151 (61.6%; quadratic-weighted κ = 0.717; Appendix C.4, Table S5).

Thank you again for your questions. We would be glad to clarify any remaining points about the scenario review or scoring rubric.

Reviewer_CFX4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds QH-BENCH, a Chinese-language benchmark for adolescent-facing content safety, and evaluates 13 open-weight Chinese-family models (Qwen2.5, InternLM2.5, GLM-4/GLM-Z1, DeepSeek, 1.5B–32B) on it. The single-turn track has 715 items organized as 10 risk domains / 50 subdomains / 143 fine-grained scenarios (5 items each), with settings drawn from Chinese institutions (youth mode, real-name verification, homeroom-teacher contact, livestream gifting, comprehensive student evaluation). The multi-turn track has 100 four-turn trajectories in a balanced 10×10 design crossing the same ten domains with ten cross-turn mechanisms (coreference, harmful composition, relational pressure, etc.). All scenarios are synthetic: GPT-5.5 generated candidates, researchers reviewed and revised. Responses are scored by claude-opus-4-8 (temperature 0, third-party endpoint) on a five-level joint safety-helpfulness scale from −2 (clearly risk-enabling) to +2 (safe and helpful), with negative scores meaning the response facilitates risk rather than merely being unhelpful. Main findings: all models score negatively on more than half of the 15 offline-contact items (pooled negative rate 79.0%), including the single-turn leader InternLM2.5-20B; relational pressure (M09) is the hardest mechanism (70% negative); model ranks still shift between tracks despite positive correlation (Spearman 0.732; InternLM2.5-20B goes 1st to 3rd, GLM-4-32B 6th to 1st); and some domain-mechanism cells fall well below what domain or mechanism averages alone would predict. A single human annotator audited 150 of the 10,595 scored units (9,295 single-turn responses plus 1,300 multi-turn trajectory scores, since the judge returns one score per four-turn interaction): 73.3% exact agreement, quadratic-weighted κ = 0.715, with the judge scoring lower than the human in 38 of 40 disagreements.

Critical strengths
  1. The gap is real and the design that fills it is sensible. Existing Chinese safety benchmarks mainly target general users, and the existing child/adolescent safety benchmarks (MinorBench, YouthSafe, KIDBench, CAREBench) are English. A benchmark where risk depends on Chinese institutional context, the user being a minor, and several turns of history is a contribution I can see the community using. Within that, the balanced 10×10 domain-mechanism design is the best idea in the paper: because each mechanism appears once per domain, mechanism effects can be compared within one benchmark instead of across studies with different items, dialogue designs, and scoring rules, which is exactly the comparability problem the introduction describes. The five-level scale is also a good choice for this population: giving refusal-only answers a 0 rather than a pass means a model cannot top the benchmark by refusing everything, and the negative levels give "safe but unhelpful" and "risk-enabling" distinct signs, which is what makes the headline finding (leaders still fail on offline contact) mean something.

  2. The reporting is careful and honest about its own limits. Bootstrap intervals accompany the means, the mechanism comparison uses a Friedman test with Holm-corrected pairwise tests, the paper says outright that M09 and M02 are not orderable from the data and that the cell-level residuals need larger samples to generalize, and the limitations section volunteers the fixed-script incoherence problem and the fact that the two tracks are not causally comparable. The negative-score framing is applied to the leaders' own results rather than only to weak models. This is above the reporting standard of most benchmark papers I review.

Critical weaknesses
  1. The automatic judge is the measurement instrument, and its validation is too thin for the quantitative claims built on it. Every headline number in the paper—the 79.0% pooled negative rate on offline contact, "every model receives negative scores on more than half" of those items, GLM-4-32B failing 6/10 relational-pressure trajectories—is a judge output. The human audit covers 150 of the 10,595 scored units (about 1.4%; 9,295 single-turn responses plus 1,300 multi-turn trajectory scores), uses one annotator (so no inter-annotator agreement exists even as a target), and shows 73.3% exact agreement with κ = 0.715. And the audit reveals a systematic bias that the paper reports but does not act on: in 40 disagreements the judge scores lower than the human 38 times (p = 1.5e-9). A judge that is systematically harsher than a human inflates negative rates, and negative rates are the paper's currency. On top of this, the audit sample was drawn from nine non-GLM models only—excluding the family that contains the multi-turn leader—with no explanation given. As it stands, I cannot tell how much of "every model fails offline contact" survives human scoring, and the abstract's numbers are stated without this caveat.

  2. The multi-turn track is too small for the mechanism-level claims, and the "culturally-grounded" label rests entirely on the authors' own judgment. Each domain-mechanism cell contains one trajectory, so each mechanism mean is built from ten items, and the RQ4 residual analysis flags specific cells on the basis of 13 model scores on a single four-turn interaction; the paper concedes this ("larger cell samples are needed") but still headlines the cells. On the data side, every scenario is GPT-5.5-generated and researcher-reviewed: no domain experts (educators, child-safety practitioners, platform trust-and-safety staff), no adolescents, and, as far as I can tell, no grounding against any external document (e.g., Chinese minor-protection regulations or platform youth-mode policies) that would tell us the taxonomy and scenario distribution reflect real risk rather than the authors' and GPT-5.5's priors. There is also no benign control track, which the authors scope out explicitly; but since level 0 exists precisely to punish mechanical refusals, without benign items there is no way to see whether a model's low scores partly reflect over-caution. The benchmark is best read as a fixed, well-organized stress test; several of the paper's stronger-sounding claims go beyond that.

Questions for authors

These are clarification requests that affect my assessment, not future-work suggestions.

  1. The audit shows the judge is systematically harsher than your human annotator (38/40 disagreements). How do the headline negative rates—especially the 79.0% offline-contact rate and the per-model 9–14 counts on those 15 items—change if you recompute them with human scores on that subdomain, or with a judge threshold shifted to remove the observed bias?

  2. Why were GLM models excluded from the human audit? GLM-4-32B is the multi-turn leader; auditing none of its responses leaves the family with the best multi-turn score unvalidated. Please audit a GLM sample.

  3. Can you add at least a second annotator on the audited sample (plus a GLM-inclusive draw) and report inter-annotator agreement? With one annotator, κ = 0.715 measures agreement with one person, not with a human standard.

  4. How often were model responses truncated at the 512-token cap, and does truncation correlate with scores? A verbose model whose advice gets cut off mid-sentence could be scored on an artifact.

  5. How sensitive are the model rankings and the mechanism ordering to the judge? Any rerun with a second judge model (or a second endpoint for the same one, given the third-party routing you flag in Appendix E) would help.

  6. For the taxonomy: was any external input used—regulatory documents, platform policy texts, educators, or practitioners? If yes, please say so in the paper; if no, please soften "culturally-grounded" accordingly.

  7. On RQ4: given one item per cell, what would you consider the minimal evidence that a flagged cell (e.g., M06×D02, R = −1.605) is a real interaction rather than item noise?

1 comment

Thank you for highlighting the value of the balanced design and for asking which findings hold under independent human scoring.

Responses to Questions

Q1. Offline contact under human scoring. Both annotators independently scored all 195 responses from all 13 models. Using the original negative-score threshold, annotator A rated more than half of the 15 items negatively for 11 models, and annotator B did so for nine. Nine models met this criterion under both sets of ratings. The pooled negative rates were 79.0% (154/195) for the judge, 61.0% (119/195) for A, and 56.9% (111/195) for B. Per-model counts ranged from 9 to 14 out of 15 for the judge, 5 to 13 for A, and 5 to 12 for B. For InternLM2.5-20B, the counts were 12, 8, and 5, respectively. Offline contact remains a weakness for most tested models. Table S4 reports the counts for each model.

Q2. GLM coverage. The original fixed-interval audit covered nine non-GLM models. The targeted audit includes all four GLM variants on offline contact, and GLM-4-32B and GLM-Z1-9B on relational pressure (M09). For GLM-4-32B, each of the three scorers rated six of ten M09 trajectories negative, with five marked negative by all three (Table S4).

Q3. Original audit and inter-annotator agreement. The original 150 units (100 single-turn responses and 50 multi-turn trajectories) were scored by one human annotator and the judge. Exact agreement was 110/150 (73.3%), with quadratic-weighted Cohen’s κ = 0.715; the judge scored lower in 38 of 40 disagreements. We did not have a second annotator rescore that full set. On the targeted 245 units, the two human annotators agreed on the negative/nonnegative distinction in 200/245 cases (81.6%) and on the exact score in 151/245 cases (61.6%), with weighted κ = 0.717 (Appendix C.4, Table S5).

Q4. The 512-token cap. The cap applies per assistant reply. The archived records do not include stop reasons or generated-token counts, so we cannot estimate the truncation rate or its association with scores.

Q5. Judge sensitivity. The human ratings test the offline-contact and GLM-4-32B M09 findings. We did not rerun the full benchmark with another judge or endpoint, so this audit does not establish the stability of model rankings or mechanism ordering.

Q6. Policy context for the taxonomy. In developing the taxonomy, we consulted public materials including China's Law on the Protection of Minors, the Regulations on the Protection of Minors in Cyberspace, and the joint guidance on regulating livestream tipping and protecting minors.

Q7. Evidence for a domain–mechanism interaction. For M06×D02, we would look for a similarly negative residual across independently written trajectories in that cell and suitable comparison cells. The trajectory is the unit of scenario replication. The current residual flags one case for inspection.

Responses to Weaknesses

W1. Validation of the main findings. Independent human scoring gave lower negative rates on offline contact (61.0% and 56.9% versus the judge’s 79.0%). For nine of the 13 models, both annotators gave negative scores on more than half of the 15 items. The revised abstract therefore describes offline contact as a weakness for most models. On relational pressure, all three scorers gave GLM-4-32B six negative scores out of ten, with five trajectories rated negative by all three (Table S4).

W2. Scope of the benchmark. Each mechanism is tested across the same ten domains, but with one trajectory per cell, RQ4 identifies cases for error analysis rather than establishing generalizable interactions. The taxonomy draws on the public materials in Q6, and researchers reviewed the scenarios for plausibility. This grounds the settings without constituting external validation. A generic refusal may earn a zero and lower the mean, but it does not count toward the negative-score rate. Without benign controls, we cannot estimate over-refusal on benign requests.

Thank you again for the detailed review. We would welcome your feedback on the revised interpretation of the audit results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Rebuttal

Dear Area Chair and Reviewers,

Thank you for taking the time to review our paper and for your constructive feedback. We have posted point-by-point responses and summarize the main changes to the manuscript below.

Paper summary. QH-Bench evaluates how language models respond to safety risks in Chinese adolescent settings. It pairs a fine-grained single-turn taxonomy with a balanced multi-turn design to distinguish which risks models miss from which cross-turn mechanisms they struggle with.

Main revisions

  • Human audit (Appendix C.4, Tables S4–S5). Two annotators independently scored 195 offline-contact responses and 50 relational-pressure trajectories. The offline-contact audit covers all 13 models, including all four GLM models.
  • Main findings (Abstract and Section 5.1). The human annotators assigned fewer negative scores on offline contact than the automatic judge did. Even so, for nine of the 13 models, each annotator assigned negative scores to more than half of the 15 items. We revised the conclusion accordingly. Under relational pressure, all three scorers gave GLM-4-32B six negative scores out of ten, with five trajectories rated negative by all three.
  • Taxonomy and scope (Sections 3.2 and 6; RQ3–RQ4). We cite the public materials that informed the risk taxonomy and clarify how to interpret the multi-turn results. With one trajectory per domain–mechanism pair, the matrix can flag specific cases for error analysis, but it does not establish generalizable interactions.

We would appreciate any feedback on the revised manuscript and would be happy to clarify any points that remain unresolved.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.