- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MSafeTemp, a multilingual safety dataset constructed through a template-based augmentation pipeline. The proposed approach extracts templates from harmful prompts, translates them into different languages, and fills the templates with harmful, benign, or culturally localized entities. The resulting dataset covers 44 languages for training and 60 languages for evaluation. Experiments on several open-weight models reveal multilingual safety gaps and show that training with the generated data can reduce attack success rates, while DPO helps mitigate over-refusal introduced by SFT.
Critical strengths
The paper discusses an important problem (multilingual safety alignment). The proposed template-based approach provides a reasonable and scalable way to construct multilingual safety data, including culturally localized harmful requests and benign counterparts. Releasing the dataset and generation pipeline could be a useful contribution to the community.
Critical weaknesses
-
The quality of the generated data is not sufficiently validated. Although the authors employ LLM-based filtering and conduct a human evaluation, the latter covers only 11 of the 60 languages, with very few annotators per language. More importantly, only 65.3% of the benign labels are endorsed in the human study. This raises concerns about the reliability of the generated data, particularly the benign examples used to evaluate over-refusal. Given that the dataset is a central contribution of this work, I would expect more thorough validation and analysis of its quality.
-
The advantages of the proposed augmentation approach over simpler alternatives remain unclear. The training experiments show that MSafeTemp improves multilingual safety, but there is no direct comparison with simpler data construction approaches, such as translating existing safety datasets or directly generating multilingual examples with LLMs. Consequently, it is difficult to determine whether the improvements come from the proposed template-based design or simply from the increased amount of multilingual safety data. Such comparisons would better establish the contribution of the proposed method.
Questions for authors
Please see points mentioned in Weakness Part.
3 comments
Official Response to Reviewer q5qT from Submission21857 Authors (2/2)
Q2: How reliable are the benign labels, given that only 65.3% were endorsed by human raters?
We agree this is an important point and looked into it more closely. All 300 benign judgments concern DIRECT items, the ones we use to measure over-refusal. Raters who did not endorse an item mostly marked it as borderline: some were unsure about the label, and others noted that the request may be harmful in particular, rare contexts.
- Human adjudication. Following the reviewer's concern, two human raters independently adjudicated all 94 benign judgments that raters marked as incorrect or unsure (90 distinct items) as benign, borderline or harmful. No item received a harmful consensus, and no item was labelled harmful by both adjudicators; their disagreements are only between benign and borderline (71% raw agreement).
| Benign judgments (n = 300) | Count | Share |
|---|---|---|
| Endorsed by expert raters | 196 | 65.3% |
| Endorsed or adjudicated benign | 257 | 85.7% |
| Adjudicated borderline | 31 | 10.3% |
| Adjudicators split (benign vs. harmful) | 2 | 0.7% |
| Skipped by raters | 10 | 3.3% |
| Adjudicated harmful | 0 | 0% |
- Do the disputed items drive over-refusal? No. Over-refusal on human-endorsed items closely tracks over-refusal on all fielded items (Llama IT / SFT / SFT + DPO: 5.7 / 5.7 / 6.7 vs. 3.2 / 6.5 / 5.6), and models refuse the disputed items no more often than the endorsed ones. This check covers the HarmBench-track items in our stored outputs; in the next revision we will extend it to the WildGuard track, release the adjudication labels with the data and mark borderline items in the benchmark.
- Training data: we screened all 201,024 benign training answers with a safety judge: 1.23% were flagged, mostly in BOUNDED rows, where a limited answer is intended. Checking all 408,816 safety rows against the response class of their prompt label, 0.35% of the full training mix violates it, and answers to harmful requests contain harmful content in only 0.07% of rows. We will remove the violating rows from the released dataset.
Q3: Can you validate the data quality more thoroughly, given that the human study covers 11 of 60 languages?
We agree that broader human coverage would strengthen the paper. We chose expert raters (AI safety, red-teaming or trust and safety experience, native or fluent speakers) because the study exposes participants to harmful content. It is IRB-approved, with 15 expert submissions over 11 languages (one or two per language) and 600 judgments of 440 distinct items; we chose depth of expertise over the number of raters. Raters found the prompts natural (4.43/5) and culturally fitting (4.46/5), and endorsed 91% of harmful labels. Inter-annotator agreement is reported in App. M (Table 57): for the four languages with two submissions, the raters agree on 136 of 160 label assessments (85%; 67.5 to 97.5% per language). Cohen's kappa is low there because nearly all items are endorsed, which is why we report both. The same ethics and welfare protocol is what limits how quickly the study can be extended, so we could not scale it within the rebuttal period.
To cover all languages, we complemented it with corpus-wide automatic checks:
| Check | Scope | Result |
|---|---|---|
| Translation quality (MetricX-24 QE, lower is better) | 337,876 template pairs, 59 languages | 4.54 (mismatched control: 15.3) |
| Severity of localized vs. generic variants (1 to 5 scale) | 2,000 matched pairs | +0.06 [0.00, +0.12] |
| Response-class check of training answers | 408,816 safety rows | 1.96% violate their class (0.35% of the full mix) |
We will list the limited human coverage in the limitations and plan to extend the study to low-resource languages in future work.
Summary. On data quality: after adjudication, 85.7% of the benign judgments are confirmed benign and none harmful, the disputed items do not drive over-refusal, and only 0.35% of the training mix fails the response-class check. On simpler alternatives: at the same data budget, translated WildGuardMix reaches the same safety but over-refuses 23 to 32 points more. We will make both results central in the revision.
We hope that our responses address the reviewer's concerns, and we are happy to engage in further discussion.
Follow-up from Submission21857 Authors
Dear Reviewer q5qT,
Thank you again for your time and your thorough review. We have uploaded a revised PDF (changes in blue) that aims to address your main concerns:
- Templates vs. more multilingual safety data (W1, Q1): Section 6 now opens its analysis with the template-free baseline (Table 5). At the same data budget and the same safety, translated WildGuardMix over-refuses 23 to 32 points more than MSafeTemp, also on the 16 unseen languages and with either translator. Paired differences with intervals are in App. L.1 (Table 68).
- Reliability of the benign labels (W2, Q2): App. M.5 reports the adjudication of all disputed benign judgments: 85.7% are endorsed or adjudicated benign, none is harmful, and the disputed items do not drive over-refusal (Tables 75 and 76). App. M.6 checks all 408,816 training answers against their response class (Table 77).
- Validation beyond 11 languages (Q3): inter-annotator agreement is in Table 74, translation-quality estimates for all 59 languages in App. M.7, and the limited human coverage is stated in Section 7. Finding human evaluators for more languages is challenging during rebuttal period, however, we are happy to work on improving the human-based signal in the next version.
As the discussion period ends soon, we wanted to check whether our response and the revision address your concerns, and we would welcome any further questions.
Best regards,
Authors
Official Response to Reviewer q5qT from Submission21857 Authors
We thank the reviewer for their review and for recognizing the importance of multilingual safety alignment and the value of releasing our dataset and pipeline. Both points raised in the review were very helpful, and we address them below. The shared tables are in the general response.
Abbreviations: HB = HarmBench, WG = WildGuard, ASR = attack success rate, OR = over-refusal on benign prompts (%).
Q1: Do the improvements come from the template-based design or simply from more multilingual safety data? How does MSafeTemp compare with translating existing safety datasets or generating multilingual examples with LLMs?
Great question! We trained Llama-3.1-8B-Instruct three times with the same recipe, the same general data and the same amount of multilingual safety data, changing only the source of the safety prompts:
The answers to all prompts come from the same generator and filter, so the only difference is the prompts themselves.
All three reach similar multilingual safety and similar capability, but the simpler alternatives refuse benign requests far more often: about 2.6 times more on HarmBench and 5.5 times more on WildGuard. Since the data volume is identical, we believe the improvement does not come from simply having more multilingual safety data, but from the template design.
Our ablation points to the reason. MSafeTemp pairs every harmful request with benign variants of the same frame, which teaches the model where the boundary lies. Removing these matched benign pairs raises HarmBench over-refusal from 13.1% to 59.4%. Notably, the translated WildGuardMix data contains benign prompts too, yet it still over-refuses at 37%.
How robust is this comparison? We checked it in three ways:
We will add this comparison in the next revision.