- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VulnSafe, a Chinese capability assessment centered on VARC (Vulnerability-Aware Response Competence) that decomposes relational safety into five complementary constructs. The paper finds disparities and distinct profiles acros LLMs and provides an approach to understanding and interpreting relational safety rather than focusing solely on refusal compliance.
Critical strengths
-
The paper tackles an important problem in the mental health and psychosocial safety space. Relational safety is very crucial, especially for cases where single-turn content in itself is not enough. Additionally, the example clearly shows the motivation, and that a warm and validating message at all times is not be the best approach.
-
I like the construct-centered benchmark design. The paper tackles this problem by grounding on existing successful methods such as VARC, and creating benchmark instances.
Critical weaknesses
-
The benchmark, in its current status is more about immediate response behaviour, but single turn analysis can be a critical limitation of the relational safety findings. Relational safety is heavily dependent on interactions, such as dependence over time, or companionship and so on. Therefore, single turn does not always show the real behaviour whether it develops, continues or reduces these risks in a conversation.
-
The human validation shows promise but the failure class precision is low. Furthermore, the individual human agreement is moderate and also the scope is small given the benchmark size.
-
Proactive should do vs prohibitive should-not comparison is interesting. But it can be confounded by scenario contents. As each scenario corresponds to one label, the comparison involves different tasks rather than matches scenarios. So comparison needs a matched analysis, or narrow the final claims about performances. (The authors do acknowledge this, but this remains a major limitation)
Questions for authors
There are some model-name inconsistencies: Section 3.4 has MiniMax-3.7 but later has MiniMax-M2.7 in eval. Appendix C.4 has GLM-5.2 but the paper shows 5.1. So, exact models, scoring rubrics, and source-to-requirement mappings are important for auditing the benchmark.
4 comments
Response to Reviewer GyjT (part 2)
Interpreting the should-do/should-not comparison.
We adopt the descriptive interpretation you suggest. The fixed classification comprises 17 should-do and 24 should-not scenarios, based on the primary behavior specified by each rubric. Figure 3 describes performance across these sampled scenario sets. It does not isolate response demand from scenario content, and we will state this directly in the interpretation of the figure.
The supplementary analysis of judge explanations further distinguishes omitted protective action from prohibited action. Its reported examples associate M11 with prohibited companionship promises and L11 and H17 with missing real-world support or crisis recognition. This distinction explains why a scene-level primary demand label need not uniquely determine the type of failure in an individual response. We will present this as an exploratory failure-type analysis, not as a content-matched test or a causal explanation of the performance gap.
The supported finding is that the current rubric-defined requirements reveal different response profiles on the benchmark. We will replace wording that attributes difficulty to an independently established proactive/prohibitive mechanism with this task-specific interpretation.
Model identities, rubrics, and source-to-requirement mappings.
The four models used as both response models and rubric judges are DeepSeek-V4-Pro, GLM-5.1, Kimi-K2.6, and MiniMax-M2.7. The appendix’s GLM-5.2 entry describes the fine-screen annotation stage, while GLM-5.1 is listed for response evaluation and judging. We will organize the model information by stage and reconcile inconsistent shorthand, including the MiniMax entry, against the original run configurations. The resulting table will trace each exact API identifier to the operation it performs.
For reproducibility, we will consolidate three types of supplementary material: a model-role table; complete rubric definitions, including applicability, required behavior, and Fail conditions; and a mapping from normative or professional source clauses to behavioral requirements, scenarios, and rubric criteria. The appendix already specifies the 41 scenario-to-construct and response-demand assignments. The source mapping will distinguish the evidence motivating a requirement from the dialogue material used to instantiate a prompt.
Finally, VARC is the response-level construct introduced in this work. Evidence-centered design is the established assessment methodology informing its operationalization into scenarios and rubrics.
References
-
Jiahe Guo, Xiangran Guo, Yulin Hu, Zimo Long, Xingyu Sui, Xuda Zhi, Yongbo Huang, Hao He, Weixiang Zhao, Yanyan Zhao, and Bing Qin. 2026. When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agents. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27309–27335.
-
Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Zuchen Gao, Fei Mi, and Lanqing Hong. 2024. CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17494–17508.
-
Yi Han, Yuanxing Liu, Weinan Zhang, and Ting Liu. 2025. Nullspace Disentanglement for Red Teaming Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 21349–21365.
-
Nicholas Meade, Spandana Gella, Devamanyu Hazarika, Prakhar Gupta, Di Jin, Siva Reddy, Yang Liu, and Dilek Hakkani-Tur. 2023. Using In-Context Learning to Improve Dialogue Safety. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11882–11910.
Thank you again for your time and consideration. We would greatly appreciate it if you could let us know whether our responses and additional experiments have sufficiently addressed your concerns. Please feel free to let us know if any further clarification is needed. We would be happy to discuss any remaining questions.
Thanks to the authors for clarifying the human audit and the scope of the comparison. I’m keeping my rating because it remains unclear what the five constructs reveal beyond performance differences across the sampled scenarios.
Response to Reviewer GyjT (part 1)
Thank you for highlighting the importance of relational safety and the value of construct-centered assessment. We address your questions by clarifying the response-level measurement target, reporting the full human-audit evidence, and making the interpretation and provenance of the diagnostic comparisons explicit.
Response-level assessment and multi-turn interaction.
VulnSafe evaluates whether a response exhibits the behavior required by a specified vulnerability-related situation. This includes recognizing crisis signals, responding to requests for exclusive companionship, and offering a route to real-world support. For example, M11 tests the immediate response to a request for continuous presence. This is a defined, observable component of relational safety; the current evaluation does not establish how a relationship develops over a sequence of interactions.
We will make that measurement target explicit and add multi-turn evaluation in future work. The proposed extension will examine escalation, repair after an earlier unsafe response, and the maintenance of boundaries across a conversation. The present results will remain scoped to the sampled response-level tasks.
Human validation and Fail-class performance.
Our human audit covers 205 responses across all 41 scenarios, with five responses per scenario and coverage across all eight evaluated models. We recruited 10 master’s students with backgrounds in psychology, ethics, or computer science, each of whom independently evaluated every response while blinded to the model’s identity. This yields 2,050 individual ratings on 205 distinct responses, providing repeated judgments for constructing the human consensus.
The scale and annotation depth can be contextualized against published safety-evaluation studies:
PS-Bench (ACL 2026) validates its automatic harmfulness detector using 160 responses, each independently assessed by two annotators, with disagreements resolved through discussion or third-annotator adjudication (Guo et al., 2026).
CoSafe (EMNLP 2024) recruits three annotators and evaluates 100 sampled responses per model for three models in its human-validation analysis (Yu et al., 2024).
Nullspace Disentanglement for Red Teaming Language Models (EMNLP 2025) recruits six annotators and computes each reported human-evaluation result from 100 sampled test cases across multiple methods and target models (Han et al., 2025).
Using In-Context Learning to Improve Dialogue Safety (Findings of EMNLP 2023) evaluates 150 examples from each dataset, collecting three human judgments per response-pair comparison and evaluation dimension (Meade et al., 2023).
These studies provide precedent for targeted human evaluations using hundreds of cases per evaluation setting. Our design emphasizes complete scenario coverage and ten independent ratings per response.
Regarding reliability, our annotators achieve Fleiss’ κ = 0.520 and 87.0% mean pairwise raw agreement. For context, Han et al. (2025) also report Fleiss’ κ = 0.52, using a three-category safety scale. We complement inter-annotator reliability with a separate evaluation against the ten-person consensus: the aggregated model judgments achieve 94.15% agreement, Cohen’s κ = 0.766, and Fail-class precision/recall/F1 of 72.73%/88.89%/80.00%, as shown in Table 11. We will foreground the sampling design, annotation depth, and class-specific results to make the evidence supporting the human audit explicit.