acceptodds
Under review as a conference paper at ICLR 2027

How I Aligned Your Model: Scaling Up Multilingual Safety Training with Templates

Abstract

Safety alignment for Large Language Models is heavily focused on English with limited multilingual representation, particularly damaging low-resource language performance. Related open training datasets are often limited in size or do not follow region-localized harmful topics to broaden the scope of alignment beyond cultures. We propose a scalable template-based augmentation pipeline to produce massive multilingual refusal and benign corpora, extracting safe templates from harmful prompts, translating them and then filling in the slots with culturally authentic entities, both harmful and benign. As a result, we release MSafeTemp, a validated corpus spanning 60 languages useful for multilingual safety alignment and evaluation. Using the dataset, we validate a strong safety and over-refusal gap in lower-resource settings, where attacks that fail entirely in English succeed once translated, while culturally localized variants expose further failures that translation alone misses. We also confirm that augmenting safety training data through templates allows for higher diversity and quality of alignment, resulting in improved safety performance beyond English. MSafeTemp and the respective pipeline are aimed at facilitating research and development of safer and more responsible models for less-represented cultures.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multilingual safety, synthetic data generation, multilingual evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_q5qTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MSafeTemp, a multilingual safety dataset constructed through a template-based augmentation pipeline. The proposed approach extracts templates from harmful prompts, translates them into different languages, and fills the templates with harmful, benign, or culturally localized entities. The resulting dataset covers 44 languages for training and 60 languages for evaluation. Experiments on several open-weight models reveal multilingual safety gaps and show that training with the generated data can reduce attack success rates, while DPO helps mitigate over-refusal introduced by SFT.

Critical strengths

The paper discusses an important problem (multilingual safety alignment). The proposed template-based approach provides a reasonable and scalable way to construct multilingual safety data, including culturally localized harmful requests and benign counterparts. Releasing the dataset and generation pipeline could be a useful contribution to the community.

Critical weaknesses
  1. The quality of the generated data is not sufficiently validated. Although the authors employ LLM-based filtering and conduct a human evaluation, the latter covers only 11 of the 60 languages, with very few annotators per language. More importantly, only 65.3% of the benign labels are endorsed in the human study. This raises concerns about the reliability of the generated data, particularly the benign examples used to evaluate over-refusal. Given that the dataset is a central contribution of this work, I would expect more thorough validation and analysis of its quality.

  2. The advantages of the proposed augmentation approach over simpler alternatives remain unclear. The training experiments show that MSafeTemp improves multilingual safety, but there is no direct comparison with simpler data construction approaches, such as translating existing safety datasets or directly generating multilingual examples with LLMs. Consequently, it is difficult to determine whether the improvements come from the proposed template-based design or simply from the increased amount of multilingual safety data. Such comparisons would better establish the contribution of the proposed method.

Questions for authors

Please see points mentioned in Weakness Part.

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer q5qT from Submission21857 Authors

We thank the reviewer for their review and for recognizing the importance of multilingual safety alignment and the value of releasing our dataset and pipeline. Both points raised in the review were very helpful, and we address them below. The shared tables are in the general response.

Abbreviations: HB = HarmBench, WG = WildGuard, ASR = attack success rate, OR = over-refusal on benign prompts (%).

Q1: Do the improvements come from the template-based design or simply from more multilingual safety data? How does MSafeTemp compare with translating existing safety datasets or generating multilingual examples with LLMs?

Great question! We trained Llama-3.1-8B-Instruct three times with the same recipe, the same general data and the same amount of multilingual safety data, changing only the source of the safety prompts:

  1. Translating an existing safety dataset: WildGuardMix harmful and benign prompts, translated into the 44 languages with a dedicated MT system (NLLB-200 3.3B).
  2. Producing the examples with an LLM: the same prompts, translated in full by Gemma-4-31B, the same model that generates our data. We consider this the closest controlled version of LLM generation, since free generation of new prompts would change the harm distribution and break the row matching.
  3. MSafeTemp: our template-based data.

The answers to all prompts come from the same generator and filter, so the only difference is the prompts themselves.

Safety dataHB ASRHB ORWG ASRWG OROR-BenchMMLUGSM8K
None (released model)16.24.710.97.54.571.384.2
(1) WildGuardMix, NLLB1.137.01.139.146.064.068.8
(2) WildGuardMix, Gemma-41.137.70.938.248.765.470.2
(3) MSafeTemp1.614.10.77.140.364.767.6

All three reach similar multilingual safety and similar capability, but the simpler alternatives refuse benign requests far more often: about 2.6 times more on HarmBench and 5.5 times more on WildGuard. Since the data volume is identical, we believe the improvement does not come from simply having more multilingual safety data, but from the template design.

Our ablation points to the reason. MSafeTemp pairs every harmful request with benign variants of the same frame, which teaches the model where the boundary lies. Removing these matched benign pairs raises HarmBench over-refusal from 13.1% to 59.4%. Notably, the translated WildGuardMix data contains benign prompts too, yet it still over-refuses at 37%.

How robust is this comparison? We checked it in three ways:

  • Uncertainty. Paired per language, MSafeTemp lowers over-refusal by 23.1 points [21.1, 25.3] on HarmBench and 32.5 points [28.4, 36.5] on WildGuard, and by 20.7 points [17.7, 23.9] on the 16 languages not in the training data. A same-recipe retrain of our model reproduces every in-house cell within about 1 point.
  • Benchmark format. Our in-house benign prompts share the minimal-pair format of our training data, which could favor MSafeTemp. On the external OR-Bench, which none of the pipelines produced, the direction is the same, with a smaller margin (5.7 and 8.4 points) that is closer to run-to-run variation on this benchmark (up to 4.6 points). We will add XSTest for all three arms.
  • Translator. Gemma-4 and NLLB give the same result (+0.7 [-0.2, +1.9] on HarmBench over-refusal).

We will add this comparison in the next revision.

AuthorsOpenReview ↗

Official Response to Reviewer q5qT from Submission21857 Authors (2/2)

Q2: How reliable are the benign labels, given that only 65.3% were endorsed by human raters?

We agree this is an important point and looked into it more closely. All 300 benign judgments concern DIRECT items, the ones we use to measure over-refusal. Raters who did not endorse an item mostly marked it as borderline: some were unsure about the label, and others noted that the request may be harmful in particular, rare contexts.

  • Human adjudication. Following the reviewer's concern, two human raters independently adjudicated all 94 benign judgments that raters marked as incorrect or unsure (90 distinct items) as benign, borderline or harmful. No item received a harmful consensus, and no item was labelled harmful by both adjudicators; their disagreements are only between benign and borderline (71% raw agreement).
Benign judgments (n = 300)CountShare
Endorsed by expert raters19665.3%
Endorsed or adjudicated benign25785.7%
Adjudicated borderline3110.3%
Adjudicators split (benign vs. harmful)20.7%
Skipped by raters103.3%
Adjudicated harmful00%
  • Do the disputed items drive over-refusal? No. Over-refusal on human-endorsed items closely tracks over-refusal on all fielded items (Llama IT / SFT / SFT + DPO: 5.7 / 5.7 / 6.7 vs. 3.2 / 6.5 / 5.6), and models refuse the disputed items no more often than the endorsed ones. This check covers the HarmBench-track items in our stored outputs; in the next revision we will extend it to the WildGuard track, release the adjudication labels with the data and mark borderline items in the benchmark.
  • Training data: we screened all 201,024 benign training answers with a safety judge: 1.23% were flagged, mostly in BOUNDED rows, where a limited answer is intended. Checking all 408,816 safety rows against the response class of their prompt label, 0.35% of the full training mix violates it, and answers to harmful requests contain harmful content in only 0.07% of rows. We will remove the violating rows from the released dataset.

Q3: Can you validate the data quality more thoroughly, given that the human study covers 11 of 60 languages?

We agree that broader human coverage would strengthen the paper. We chose expert raters (AI safety, red-teaming or trust and safety experience, native or fluent speakers) because the study exposes participants to harmful content. It is IRB-approved, with 15 expert submissions over 11 languages (one or two per language) and 600 judgments of 440 distinct items; we chose depth of expertise over the number of raters. Raters found the prompts natural (4.43/5) and culturally fitting (4.46/5), and endorsed 91% of harmful labels. Inter-annotator agreement is reported in App. M (Table 57): for the four languages with two submissions, the raters agree on 136 of 160 label assessments (85%; 67.5 to 97.5% per language). Cohen's kappa is low there because nearly all items are endorsed, which is why we report both. The same ethics and welfare protocol is what limits how quickly the study can be extended, so we could not scale it within the rebuttal period.

To cover all languages, we complemented it with corpus-wide automatic checks:

CheckScopeResult
Translation quality (MetricX-24 QE, lower is better)337,876 template pairs, 59 languages4.54 (mismatched control: 15.3)
Severity of localized vs. generic variants (1 to 5 scale)2,000 matched pairs+0.06 [0.00, +0.12]
Response-class check of training answers408,816 safety rows1.96% violate their class (0.35% of the full mix)

We will list the limited human coverage in the limitations and plan to extend the study to low-resource languages in future work.

Summary. On data quality: after adjudication, 85.7% of the benign judgments are confirmed benign and none harmful, the disputed items do not drive over-refusal, and only 0.35% of the training mix fails the response-class check. On simpler alternatives: at the same data budget, translated WildGuardMix reaches the same safety but over-refuses 23 to 32 points more. We will make both results central in the revision.

We hope that our responses address the reviewer's concerns, and we are happy to engage in further discussion.

AuthorsOpenReview ↗

Follow-up from Submission21857 Authors

Dear Reviewer q5qT,

Thank you again for your time and your thorough review. We have uploaded a revised PDF (changes in blue) that aims to address your main concerns:

  • Templates vs. more multilingual safety data (W1, Q1): Section 6 now opens its analysis with the template-free baseline (Table 5). At the same data budget and the same safety, translated WildGuardMix over-refuses 23 to 32 points more than MSafeTemp, also on the 16 unseen languages and with either translator. Paired differences with intervals are in App. L.1 (Table 68).
  • Reliability of the benign labels (W2, Q2): App. M.5 reports the adjudication of all disputed benign judgments: 85.7% are endorsed or adjudicated benign, none is harmful, and the disputed items do not drive over-refusal (Tables 75 and 76). App. M.6 checks all 408,816 training answers against their response class (Table 77).
  • Validation beyond 11 languages (Q3): inter-annotator agreement is in Table 74, translation-quality estimates for all 59 languages in App. M.7, and the limited human coverage is stated in Section 7. Finding human evaluators for more languages is challenging during rebuttal period, however, we are happy to work on improving the human-based signal in the next version.

As the discussion period ends soon, we wanted to check whether our response and the revision address your concerns, and we would welcome any further questions.

Best regards,

Authors

Reviewer_5PTpOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a template-based pipeline for producing multilingual, culturally localized safety data at scale and releases MSAFETEMP, the corpus it generates. An English harmful seed (200 HarmBench behaviours for evaluation; 2,500 WildGuardMix prompts for training) is abstracted by an LLM (Gemma 4 31B IT) into a template with typed harm-bearing slots, the template is natively translated with slots intact, two paraphrases are added, and three branches then fill the slots: generic harmful entities (TR), benign minimal-pair entities (OR, split into DIRECT/BOUNDED), and entities drawn from manually verified per-language "cultural cards" (PH). Every stage is gated by per-criterion Boolean LLM judges (GPT-OSS-120B), and target responses are judged by GPT-OSS-Safeguard-120B. The result is 1.8M WildGuard-track rows in 44 languages and 217K HarmBench-track rows in 60 languages, with provenance-tracked, leakage-safe splits. Seven open models are evaluated on EN/TR/PH harmful prompts and DIRECT benign pairs; translated variants break Olmo-3.1-32B (0.0% → 13.7%) and both Apertus models far more often than English, while Gemma 4 and Llama-3.3-70B are actually less often broken outside English. PH prompts lower ASR for 5/7 frontier models in a paired per-language test but break a partially different set of behaviours (Jaccard 0.40–0.45 with TR). SFT on MSAFETEMP refusals + benign answers + Dolci general data, followed by on-policy DPO, sharply reduces multilingual ASR for Apertus-8B, Olmo-3-7B and Llama-3.1-8B (including on 16 held-out languages), at the cost of increased over-refusal (partly recovered by DPO) and a capability drop that is largest on GSM8K. Supporting material includes entity-diversity analysis, cluster-bootstrap CIs, Holm-corrected paired tests, an independent-judge check on 5,000 responses, a 15-respondent IRB-approved expert study over 11 languages, and a datasheet with a gated release plan.

Critical strengths
  1. A genuinely scalable, well-engineered resource with controlled variants. Harmful, benign and localized prompts share the same template, so over-refusal and attack success are measured on matched content, and the split-by-seed design prevents leakage. The diversity analysis (D/F ≈ 0.70, no entity > 1% of a language's fills) addresses the obvious objection to template augmentation. Few multilingual safety datasets reach 60 languages with this level of provenance, and the release plan (gated harmful rows, every harmful prompt paired with a refusal, cultural cards and pipeline prompts open) is responsible. This is the main reason the paper merits acceptance.

  2. Evaluation breadth and honesty. Seven frontier open models on two tracks, three training families, held-out-language transfer (Table 27), an independent judge check, and explicit reporting of results that cut against the narrative (PH lower than TR for most models; paraphrases help on WildGuard but not HarmBench; benign labels endorsed by only 65% of native experts).

Critical weaknesses
  1. The training claim attributed to templates is never isolated from plain multilingual refusal SFT. The title and RQ3 are about "scaling up multilingual safety training with templates", and the abstract says the work "confirms that augmenting safety training data through templates allows for higher diversity and quality of alignment". The ablations (Table 3, Table 24) compare base/IT checkpoints against MSAFETEMP SFT (+DPO) only. There is no control that fine-tunes the same checkpoints on the same 2,000 seeds machine-translated into the same 44 languages with the same Gemma-generated refusals and the same Dolci mix, nor a TR-only versus TR+PH versus TR+OR data ablation. Without these, the observed ASR reductions cannot be attributed to templates, paraphrases or cultural fills rather than to "any multilingual refusal data"; prior work (e.g. RefusEU, PolyGuard-style data) already shows the latter helps. For a paper whose contribution is the data-generation method, this is the single most important missing experiment. The capability cost also looks larger than the text suggests: reading Table 24, Llama-3.1-8B-Instruct appears to lose roughly 7 MMLU points and 16 GSM8K points after MSAFETEMP SFT, and its OR-Bench refusal rate rises from 4.6% to 35.7% (24.8% after DPO); the main text characterises the cost as "modest on MMLU" based on Apertus alone.

  2. The low-resource safety gap, the paper's headline evaluation result, rests on an LLM judge whose validity in low-resource languages is never checked. ASR in Hausa, Yoruba, Swahili, Nepali and similar languages is defined as GPT-OSS-Safeguard-120B's verdict of "unsafe compliance", a judge that the independent check (App. I) shows agrees only moderately with other judges on attack success (κ = 0.40–0.46) and that "also counts dual-use or partially compliant answers". App. I is not stratified by resource tier, and the human study (App. M) covers only prompts, not judged responses, in 11 languages that are all European plus Simplified Chinese; none of the low-resource African or South Asian languages where the gap is claimed is covered. A judge that reads a degraded or off-topic low-resource response as "partial compliance" would manufacture exactly the pattern reported. Relatedly, the claim that localization "exposes failures that translation alone misses" (Jaccard 0.40–0.45 between TR and PH failure sets) has no null baseline: the paper's own Appendix K.1 shows that within a template the entity decides the outcome, so two different generic fills of the same templates would also disagree on which behaviours break. Without the TR-vs-TR Jaccard, the "∼45% PH-only failures" statistic cannot be interpreted as a property of cultural grounding.

Questions for authors
  1. Can you add the translation-only control: SFT (+DPO) on the same seeds machine-translated into the same languages with the same refusal responses and general mix, and a TR-only / TR+PH / TR+OR data ablation? Does MSAFETEMP still win on multilingual ASR, over-refusal and capability retention?

  2. Please report judge–human agreement on responses for at least a handful of low-resource languages (e.g. Hausa, Swahili, Yoruba, Nepali), separately for "unsafe compliance" and "refusal", and report App. I's κ by resource tier. What fraction of low-resource "unsafe compliance" verdicts are partial, degenerate, or off-topic responses?

  3. What is the Jaccard overlap between the failure sets of two different generic (TR) fills of the same templates, matched in count to the PH arm? Is the 0.40–0.45 TR–PH overlap lower than that baseline?

  4. Please define "Orig." (Table 2) versus "EN" (Table 16/26) and explain why Llama-3.3-70B is 13.0% in one and 8.0% in the other.

  5. Given that native experts endorsed only 65% of BENIGN labels, how many DIRECT benign training rows (120K) are likely to be harmful requests paired with helpful answers, and were the SFT answers to those rows screened for unsafe content?

  6. Gemma 4 31B IT generated every prompt and every refusal response and is also an evaluated target. Did you check whether the generator's own ASR is deflated by self-generation (e.g. by evaluating it on prompts generated by a different model)?

  7. Prior work on culturally adaptive red-teaming benchmark generation (e.g. CAGE, ICLR 2026) and the Southeast-Asian/Latin-American culturally adapted red-teaming studies address the same localization problem and should be discussed, ideally as a comparison point for the PH branch.

4 comments
AuthorsOpenReview ↗

Official Response to Reviewer 5PTp from Submission21857 Authors

We thank the reviewer for a very careful and precise review, for positively recognizing MSafeTemp as a well-engineered resource with controlled variants, and for valuing our transparent reporting. Several of the new experiments follow directly from the reviewer's suggestions. We address each point below, using the reviewer's numbering; shared tables are in the general response.

Abbreviations: HB = HarmBench, WG = WildGuard, HB-JB = jailbreak-wrapped HarmBench, ASR = attack success rate, OR = over-refusal (%).

W1 / Q1: Can the training gains be attributed to templates rather than to any multilingual refusal data?

We ran the translation control the reviewer described, with two small changes that we believe make it stricter:

  • We translate WildGuardMix's full pool (47,851 unique prompts) instead of only our 2,000 seeds, which gives the baseline more distinct prompts than our own data.
  • The baseline uses WildGuardMix's own benign prompts as benign training data.

Everything else matches: the same Llama-3.1-8B-Instruct checkpoint, the same general data, row-matched safety rows per language and class (105,916 vs. 105,983), the same response generator and judge filter, and the same recipe and seed. The three outcomes the reviewer asked for:

MSafeTemp minus template-free (NLLB), points [95% CI]HB ASRHB ORWG ORHB-JB OR
All languages+0.6 [+0.4, +0.8]-23.1 [-25.3, -21.1]-32.5 [-36.5, -28.4]-33.1 [-35.4, -30.9]
16 languages unseen in training+0.4 [-0.1, +0.8]-20.7 [-23.9, -17.7]n/a-24.9 [-27.2, -22.7]

Differences are paired per language and averaged, so they can differ slightly from the difference of the means in the general-response table.

On external benchmarks, OR-Bench gives 40.3 vs. 46.0 (smaller than the in-track gap, but the same direction), AyaRT 0.5 vs. 0.4 and PolyTox 6.2 vs. 7.0. Capability is equal (MMLU 64.7 vs. 64.0, GSM8K 67.6 vs. 68.8). Using Gemma-4 as the translator instead of NLLB does not change this.

We also report the earlier control that translates the English counterparts of our own template rows with NLLB, which keeps the minimal-pair structure and isolates in-language generation. Here the two are close: ASR -0.9 [-1.1, -0.7] in our favor over all languages and -1.5 [-2.0, -0.9] on unseen languages, while over-refusal is mixed (HB OR 14.1 vs. 10.5, WG OR 7.1 vs. 9.9, OR-Bench 35.7 vs. 34.0). Taken together, the two controls say that the benefit of the template design comes mainly from the minimal-pair structure, and only to a small degree from generating in each language. For reference, a same-recipe, same-seed retrain of our model differs by +0.0 [-0.1, +0.2] over all languages and +0.2 [-0.2, +0.6] on unseen languages. As the reviewer may anticipate, this is a reproducibility check rather than seed variance, so we do not claim the margin on the four lowest-resource languages (-2.7, with a retrain difference of +1.1 there).

Q1 (ablation)

The cumulative ablation (general response) shows that harmful data alone gives 59.4% HB and 71.1% WG over-refusal, and the matched benign pairs bring this to 13.1% and 6.9%. The localized and jailbreak-wrapped rows mainly lower over-refusal on wrapped prompts (20.5 to 11.0) and on OR-Bench (42.7 to 35.7); on HB their effect is small. We will credit the training benefit to the matched benign pairs in Section 6, as the reviewer suggests.

W1 (capability)

Thank you for reading Table 24 this closely. The reviewer is right about Llama, and "modest on MMLU" holds only for some starting points. Across families, including Apertus-v1.5-8B, which we add in the revision, the final model compares with the checkpoint it starts from as follows (points):

Start, then + MSafeTemp SFT + DPOMMLUGSM8KBBHIFEval
Apertus-8B, IT+0.1-11.9-5.2-0.9
Apertus-8B, vendor SFT+1.7-3.9-4.9+10.0
Apertus-v1.5-8B, IT-6.2-7.8-7.9-11.3
Apertus-v1.5-8B, vendor SFT-2.2-4.6-5.3-3.5
Olmo-3-7B, IT-4.8-6.8+8.2-5.1
Olmo-3-7B, vendor SFT-1.2-2.5-0.1+0.8
Llama-3.1-8B, IT-8.4-15.5-11.8-0.8

The cost is largest on top of a released instruct model, and small when MSafeTemp enters post-training at the vendor SFT stage (MMLU within about 2 points, GSM8K -2.5 to -4.6). GSM8K is the most consistent cost. For Llama, general data alone costs as much MMLU (65.4 vs. 65.2) and most of the GSM8K loss, while MSafeTemp alone costs 6.9 MMLU and 8.5 GSM8K points (general response, Q2). We attribute this to Llama's originally stronger alignment which is degraded by any weaker open SFT mixes, including Dolci; Llama 3.1 does not open source training data or intermediate checkpoints to test with. We will replace "modest on MMLU" with this table in the main text.

AuthorsOpenReview ↗

Official Response to Reviewer 5PTp from Submission21857 Authors (2/3)

W2 / Q2: Is the judge valid in low-resource languages, and could degenerate replies create the gap?

Thank you for raising this; we checked it in three ways.

First, Table 38(b) in App. I reports agreement by resource tier. As the reviewer notes, it is lower in the low-resource tier (0.31 for Llama Guard 4, 0.42 for Llama-3.3-70B) than in English, and it is computed on our 8B training checkpoints. We will reference it from the main text and state this scope.

Second, the concern about degenerate replies is well founded for the leading-model panel, since the HarmBench runs in Table 2 predate our degenerate-response filter. We therefore applied the filter to the stored responses of the whole panel:

ModelTR ASR off / onPH ASR off / onValid replies, low-resource tier
Qwen3.5-27B0.4 / 0.40.1 / 0.194.3%
gpt-oss-20b0.4 / 0.40.3 / 0.281.1%
Gemma 4 31B4.2 / 4.22.5 / 2.599.2%
Olmo-3.1-32B13.7 / 13.811.9 / 12.096.5%
Llama-3.3-70B5.3 / 5.12.6 / 2.279.6%
Apertus-8B10.1 / 9.712.2 / 11.452.3%
Apertus-70B13.0 / 12.56.0 / 5.050.7%

No ASR cell moves by more than 1.0 point. The share of unsafe verdicts that fell on replies the filter removes is at most 2.7% for four models, 7.3% for Llama-3.3-70B, and 14.0% and 16.8% for the two Apertus models, which are also the models with the lowest valid-reply shares in low-resource languages.

Third, we were not able to collect human judgments of responses in low-resource languages within the rebuttal period. We therefore scope the claim: the paper will report valid-reply shares per language and training stage, and state that per-language values in low-resource languages depend on a judge that is not validated there. The finding we keep is that English results do not predict multilingual behavior, in either direction.

Q3: What is the TR-vs-TR Jaccard null?

Thank you for this suggestion! We drew two disjoint generic fill sets of the same templates, each matched in size to the localized arm, over 500 resamples, with cluster-bootstrap intervals over behaviors.

SettingGeneric vs. genericGeneric vs. localizedDifference [95% CI]
Wording fixed, Apertus-8B (12 languages)0.4810.4550.026 [-0.028, 0.081]
Wording fixed, Olmo-3.1-32B (12 languages)0.4730.3980.075 [0.029, 0.121]
Table 2 localized arm, 8 models, 58 languages0.3880.2930.095 [0.074, 0.119]

The 8 models are the 7 of Table 2 plus Apertus-v1.5-8B.

With the wording fixed, the localized-only share of one-arm failures is 0.465 and 0.436, close to the 0.50 expected by symmetry, so the local entity alone behaves like a second generic sample. For the Table 2 arm, the plain overlap is clearly lower for 6 of 8 models. Since localized prompts break fewer behaviors for most models, which lowers Jaccard by itself, we also computed a rate-robust measure (does the generic arm break the cells the localized arm breaks). Under it, the difference remains significant for 3 models (Gemma 4, Olmo-3.1-32B, Apertus-8B). We will therefore drop "failures that translation alone misses" and the "about 45% of arm-specific failures" statistic, and write that the localized arm reaches failure sets that differ from generic resampling for some models, without attributing this to the entity alone.

AuthorsOpenReview ↗

Official Response to Reviewer 5PTp from Submission21857 Authors (3/3)

Q4: How are Orig. and EN defined, and why does Llama-3.3-70B score 13.0 in one and 8.0 in the other?

Orig. is the 200 untouched HarmBench seeds; EN is the English template arm with generated fills. Llama-3.3-70B is broken more often by the original seeds (13.0, over 200 prompts) than by the filled English variants (8.0). We define both in a new notation block. Following the reviewer's remark, we also recompute the factor in Section 5 against the matched EN arm: it becomes 1.7 to 3.3 instead of 2.4 to 4.4, and we report the matched version.

Q5: How many DIRECT benign training rows may be harmful requests paired with helpful answers, and were the answers screened?

We screened the benign answers with a safety judge, first in the Llama and Olmo training mix and then in the full training mix:

Benign arm, flagged unsafeLlama/Olmo mixFull mix
BOUNDED11.1%10.9%
Jailbreak-wrapped2.0%2.1%
DIRECT0.64%0.59%
Short contrast0.32%0.35%
Vanilla0.12%0.14%
All0.97%1.23%

The per-arm rates are stable between the two. Prompted by the reviewer's question, we also checked all 408,816 safety rows against the response class of their prompt label: harmful requests should be refused, and benign requests answered without harmful content. Answers to harmful requests contain harmful content in only 0.07% of rows, and benign requests are fully refused in 0.37%. The most common violation is a harmful request answered with compliance that the judge does not rate as harmful (2.2%). In total, 1.96% of safety rows violate their class, which is 0.35% of the full SFT mix. We will remove these rows from the released dataset. BOUNDED requests are ones where a careful, limited answer is appropriate, so a higher flag rate is expected; removing the flagged answers keeps only those that stay within bounds. All disputed human judgments concern DIRECT items. Two human raters adjudicated all 94 of them: none was judged harmful, and 31 (10.3% of the 300 benign judgments) were judged borderline. We will mark these items in the release and state in the limitations that some DIRECT rows are borderline requests with helpful answers.

Q6: Is Gemma's own ASR deflated by self-generation?

We rebuilt 50 HarmBench seeds in 11 languages with Llama-3.3-70B as the generator and compared Gemma-4-31B against three reference models on both versions.

Gemma advantage (points)Gemma-built promptsLlama-built promptsChange [95% CI]
TR ASR-3.1-2.6+0.6 [-1.0, +2.1]
Over-refusal-8.1-3.5+4.6 [+0.2, +9.6]

There is no self-bias on attack success. Gemma's over-refusal advantage does shrink, and we will add a note to Table 2 so that over-refusal comparisons involving Gemma are read with this in mind.

Q7: Related work on culturally adaptive red-teaming

Thank you for the pointers. In the next revision, we will discuss CAGE and the Southeast-Asian and Latin-American culturally adapted red-teaming studies in Section 2, together with the other works suggested by the reviewers.

Presentation. We will add a notation block, a language inventory table (family, resource tier, train/eval membership) and a restructured Section 6.

We hope these results address the reviewer's concerns, and we would be glad to discuss any remaining questions.

AuthorsOpenReview ↗

Follow-up from Submission21857 Authors

Dear Reviewer 5PTp,

Thank you again for your valuable review and overall positive score. We have submitted the new revision of our work and the revised PDF (changes in blue) includes the controls you proposed:

  • Attribution to templates (W1/Q1): the translation control, the MT-of-templates control and the component ablation are in Section 6 (Table 5), with paired differences in App. L.1 (Table 68). We credit the benefit to the matched benign pairs, as you suggested.
  • Capability (W1): "modest on MMLU" is replaced by the capability change per family and starting checkpoint (Table 4, main text).
  • Judge validity and degenerate replies (W2/Q2): the validity gate on the full Table 2 panel is in App. D.4 (Table 32), judge agreement by tier in Table 50(b), and valid-reply shares per language and training stage in App. E.3 (Tables 42 and 43).
  • Jaccard null (Q3): App. F.1 (Table 45), with the scoped claim in Section 5.
  • Q4: Orig. and EN are defined in Section 4 and Table 2; the matched factor (1.7 to 3.3) is in App. K.
  • Q5: the screen of all training answers against their response class is in App. M.6 (Table 77).
  • Q6: the generator-overlap check is in App. D.5 (Table 33), noted in Section 5.
  • Q7: the suggested related work is now referenced in Section 2, and the language inventory in Table 10.

Table numbers in our response refer to the submission (Table 24 is now Table 27, Table 38 is Table 50). We once again thank the reviewer for their useful suggestions which helped us improve our paper. As the discussion period is coming to an end, we wanted to check whether our response and the revision address your concerns, and we would be glad to discuss any remaining questions.

Best regards,

Authors

Reviewer_HtqMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Pipeline: template-based augmentation turns English harmful seeds into typed-slot templates, translates them, and refills the slots three ways: generic harmful (TR), benign minimal pairs (OR, DIRECT/BOUNDED), and culturally localized fills (PH) drawn from per-language "cultural cards".
  • Data: MSafeTemp, ~1.8M WildGuardMix-derived prompts in 44 languages (train/test split by seed) plus a 217k HarmBench evaluation track in 60 languages.
  • Evaluation: 7 open models; several show a gap between translated and English ASR.
  • Training: SFT→DPO on Apertus, Llama-3.1-8B and Olmo-3-7B reduces multilingual ASR, at some cost in over-refusal and capability.
Critical strengths
  • Motivation: there are many multilingual safety benchmarks but comparatively little work on multilingual safety alignment. This paper contributes training data and training experiments, not only another benchmark.
  • Pipeline design: the four requirements R1–R4 (harm fidelity, linguistic correctness, cultural authenticity, over-refusal control) are sensible, and the pipeline addresses each of them. Abstracting prompts into templates avoids refusals from the generator model, and almost all templates pass validation (≈99.5%).
  • The training study (§6) is the most novel and interesting part of the paper. It goes beyond evaluation and shows that the data improves safety when used for training:
    • SFT on MSafeTemp sharply cuts multilingual ASR on both tracks for the instruction-tuned checkpoints of all three families.
    • On-policy DPO then recovers much of the over-refusal that SFT introduces on the minimal-pair OR metric.
    • Applying the data at different stages (base, vendor SFT, IT) gives a useful view of when safety data should enter the pipeline.
  • Culturally aware slot swapping plus matched harmful/benign minimal pairs:
    • Controlled comparisons share frame, seed and judge.
    • Separating benign controls into DIRECT and BOUNDED is useful, since refusing a BOUNDED request is not clearly wrong.
  • The human study is handled with unusual care for research ethics. In my experience, safety and red-teaming papers that expose annotators to harmful content rarely report IRB review, recruitment criteria or annotator welfare. This paper reports all of it (App. M):
    • IRB approval for a study that exposes participants to harmful synthetic content.
    • Recruitment: expert informants only, with experience in AI safety, red-teaming or trust & safety. Crowdworkers, the general public, and anyone in a dependency relationship with the investigators are explicitly excluded.
    • Informed consent: an information sheet, a content warning naming the harm categories, and six explicit consent statements. Participants can withdraw or request deletion of their data up to anonymisation.
    • Exposure mitigation: a 40-item / ~30-minute session cap, the option to skip or stop at any time, and a pointer to counselling services.
    • Privacy: no name, e-mail or IP is collected, and contact details are stored separately from ratings.
    • Compensation stated openly: participation was unpaid, voluntary and unconnected to employment.
    • Full instructions and rating rubric are reproduced.
    • I would like to see more papers in this area follow this example.
  • Sound evaluation methodology:
    • seed-level train/test split with no leakage across paraphrases or languages
    • cluster-bootstrap CIs and paired tests
    • an independent cross-family judge check
    • a thorough entity-diversity / mode-collapse analysis
  • Held-out-language transfer: after training, ASR on the 16 languages never seen in training approaches ASR on seen languages.
  • The limitations section is open about the main weaknesses.
Critical weaknesses

Claims and positioning

  • RQ1 / contribution 3 overclaim. That English safety does not transfer to other languages, worst for low-resource ones, is well established. The paper itself cites this in Related Work; see also [4], [7], [16]. Presenting it as a research question and a contribution makes §5 look more novel than it is.
  • Two claims in the abstract are stronger than the evidence:
    • "Higher diversity and quality of alignment" from templates: App. C shows diversity within the corpus, but nothing is compared against a non-template alternative.
    • Localized variants "expose further failures": this means different failures, not more (App. F: ASR on par with generic fills, Jaccard 0.40–0.45). The controlled evidence covers only 2 models × 12 languages (Tab. 34), and aggregate PH ASR is lower than TR for most models.

Language coverage

  • The abstract conflates the two tracks: "60 languages" is the HarmBench evaluation track; training covers 44. The abstract should state both.
  • The language list is buried and Indo-European-centric:
    • There is no consolidated language inventory (family, resource tier, train/eval membership); it has to be pieced together from appendix tables.
    • 40 of the 60 evaluation languages are Indo-European, mostly European.
    • Chinese is the only Sino-Tibetan language.
    • Sub-Saharan Africa gets 5 languages from 2 families (Amharic, Hausa, Swahili, Yoruba, Zulu). Only 3 of them are in training, and there is no Nilo-Saharan language.
    • There are no Dravidian languages and no indigenous languages of the Americas.
    • The paper should list these gaps in its limitations.

Data quality and validation

  • Translation quality is only evaluated indirectly:
    • The fluency and naturalness of the final prompts are checked by an LLM judge and rated by humans for 11 languages, so translation quality is captured implicitly.
    • Translation itself, i.e. the template translation step and whether meaning and harm survive it (R1), is never evaluated directly. There is no MT metric, no back-translation, and no human check of the African languages highlighted in the results.
    • Among the languages humans did rate, Greek (3.38) and French (3.64) scored lowest on naturalness (Tab. 56).
  • The cultural layer is heavily LLM-driven:
    • Cards are generated by Gemma 4 31B and checked by Gemini Flash 3.6.
    • The extent of "manual curation" is not reported, and the entity pools are small: 19–68 entities per language (Tab. 13).
    • Zulu has no card, and no reason is given.
    • The pools for the 15 HarmBench-only languages were regenerated after the run, so their PH fills cannot be checked against the released cards (App. C, footnote 2).
  • The human study's coverage is limited:
    • 11 languages with 1–2 raters each, and no rater demographics.
    • All 11 are European languages or Chinese. No African or South Asian language was rated, so none of the languages emphasized for the largest gaps is validated.
    • Agreement is hard to interpret. κ = 0 for English and German reflects near-unanimous endorsement (97.5% raw agreement). Ukrainian (67.5% raw, κ = 0), however, shows real disagreement (Tab. 57).
  • Label validity is uncertain:
    • Only 65% of BENIGN labels were endorsed (19% disputed, 12% unsure; Tab. 54), and there is no DIRECT vs. BOUNDED breakdown. The OR results may therefore overstate true over-refusal, as the authors acknowledge.
    • Harm labels are inherited from the English seeds, although PH substitutions can raise severity.

Evaluation (§5)

  • Few new insights. The section mostly confirms the translated-vs-English gap already reported in prior work.
  • The model panel misses frontier models:
    • No closed models.
    • No Kimi, MiniMax, Mistral, GLM or DeepSeek.
    • Llama 3.3 is used rather than Llama 4.
    • The largest model is 70B, yet "frontier" covers Apertus-8B.
  • Adversarial prompting is limited to fixed wrappers:
    • The wrapper bank is applied to 200 matched families and to wrapped HarmBench prompts on a wider panel that includes trained checkpoints (App. J).
    • There are no adaptive or optimization-based attacks (GCG/PAIR/TAP), no code-switching, and no multi-turn attacks.
    • Related Work does not discuss attacks.
  • Generator overlap and judge reliability:
    • Gemma 4 31B generates the prompts and the SFT responses and is also an evaluated target. gpt-oss-20b shares a family with the judge.
    • Agreement on ASR with independent judges is κ = 0.40–0.46 overall and 0.31–0.42 in the low-resource tier (Tab. 38). There is no per-language breakdown and no human validation of the responses.
    • Low ASR in low-resource languages may reflect failure to comprehend rather than safety (cf. the degenerate base-model replies in Tab. 3; [7]; see [8] for methodological context).

Training (§6)

  • The training section is only about one page, although it is the paper's most interesting part. I would have liked more analysis and ablations here, for example less space on §5 and more on training.
  • No non-template or component-level training ablation isolates what the corpus contributes. Stage comparisons and the entity-vs-rephrasing control (App. F) exist, but there is:
    • no data baseline: machine-translated WildGuardMix, English-only data, or RefusEU [6];
    • no comparison to alignment methods [5];
    • no Dolci-only control (Dolci has a 73.5% blend weight and makes up 81.9% of SFT rows);
    • no training ablation of the TR, PH, benign, BOUNDED or wrapped data.
  • The trade-offs for the other model families belong in the main text. Table 3 shows only Apertus; the other families are in App. D.2. From IT to SFT+DPO (Tab. 24/25):
    • Llama: OR-Bench 4.6→24.8, MMLU 72.0→63.6, GSM8K 84.0→68.5. The capability cost being "modest on MMLU" does not hold for Llama.
    • Olmo: multilingual OR on HarmBench 2.7→19.8.
    • DPO does not consistently fix OR-Bench: it rises from 27.4→34.6 for Apertus and 26.4→49.8 for Olmo after DPO.
    • External toxicity results are inconsistent, unlike the in-track ASR reductions. PolyTox: Apertus 4.7→5.8, Llama 4.4→6.1, Olmo 7.0→5.7.
  • Cross-family comparisons are confounded: Llama and Olmo use a 400k subsample with a different training stack than Apertus (App. L). All runs use a single seed.

Literature and citations

[1] XL-SafetyBench (Choi et al., 2026): country-grounded adversarial prompts plus a cultural-sensitivity track. Its Neutral-Safe Rate separates refusal from non-engagement. This is the prior work closest to the PH branch. [2] SEA-Guard (Tasawong et al., ACL 2026): agentic generation of culturally grounded safety (safeguard) data. Only its companion benchmark is cited. [3] WildJailbreak (Jiang et al., NeurIPS 2024): contrastive harmful/benign training pairs. Matched benign controls for safety training already exist in English, so this should be discussed. [4] M-ALERT (Friedrich et al., 2024): 75k evaluation prompts in 5 languages. Relevant context for the claim that open datasets are limited in size; [3] is the stronger counterexample on training scale. [5] Multilingual safety-alignment methods, missing as RQ3 baselines (one representative baseline would suffice): MPO (ACL 2025), MLC/"Align Once" (2026), Multilingual Self-Distillation (2026), shared safety neurons (2026), sparse weight editing (2026), layer swapping (EACL 2026), Soteria (EMNLP 2025). [6] RefusEU (Krasnodebska et al., 2026): already shows the RQ3 finding for 12 EU languages, but is mentioned only in passing. [7] TukaBench (Akinode et al., 2026): culturally adapted prompts get the least refusal, a trend that contrasts with PH (different models, languages and outcome definitions). It also shows judge reliability dropping in low-resource languages. [8] Malik et al. (2026), "safety-by-failure" (multimodal models): low ASR in low-resource languages can come from failure to comprehend. [9] RabakBench (Chua et al., 2025): an LLM localization pipeline that overlaps the proposed pipeline (it also covers guardrail evaluation). [10] IndicSafe (2026): culturally grounded Indic prompts and over-refusal in low-resource languages. [11] BabelSafe (2026): region-specific cultural injection during generation. [12] SafeWorld (Yin et al., NeurIPS 2024): geo-diverse safety benchmark plus DPO training. [13] CompositeHarm (Shukla et al., 2026): harm fidelity under translation (R1). [14] Safety-Tuned LLaMAs (Bianchi et al., ICLR 2024): safety SFT raises over-refusal (an English-language precedent). [15] Response-based KD for multilingual jailbreak prevention (Zhang et al., 2025): a similar setup (SFT on teacher refusals, including boundary cases) but the opposite outcome, higher ASR, plus a GSM8K drop. It motivates an ablation of the BOUNDED data. [16] For completeness: Multilingual Blending, Atil et al. 2025, the Krasnodebska et al. SLR, Banerjee et al. 2026, Verma & Bharadwaj 2025, LSR, SORRY-Bench, Multilingual Collaborative Defense.

Automated literature review also found the following:

Errors in existing references:

  • The Gemini reference is wrong: wrong team, wrong year, a garbled note, and it is used to cite "Flash 3.6".
  • Outdated venues for Wang'24, Yong'25, PolyGuard and Gehman.
  • MMLU, GSM8K, IFEval and BBH are not cited.

Additionally:

  • Llama-Guard-4 is not cited.
  • Qwen3.5 appears to be cited via the Qwen3 report.
  • Dolci-Instruct-SFT has no citation.
Questions for authors
  • Why does Zulu have no cultural card? Is it data scarcity, failed generation, or something else?
  • Beyond the fluency of the final prompts, can you evaluate the translation step directly for a sample of low-resource languages (COMET/MetricX, back-translation, or native-speaker review of meaning and harm preservation)?
  • Manual verification of cultural cards: who did it, for which languages, and what share of entries were edited or removed?
  • Do you plan to extend the human study to African and other low-resource languages? Can you add rater demographics?
  • How well does GPT-OSS-Safeguard agree with human judgments of responses, per language? What share of frontier-model responses are non-comprehension or degenerate in Hausa, Yoruba and Zulu?
  • What do you get from SFT on (a) machine-translated WildGuardMix without templates, and (b) Dolci only, at a matched budget? Can you add a training ablation of the PH/benign/BOUNDED/wrapped components?
  • Can you compare against at least one multilingual alignment method ([5]) or against RefusEU data?
  • Why Llama 3.3 instead of Llama 4? Can you add Kimi, GLM, Mistral, MiniMax, DeepSeek, and at least one closed frontier model?
  • App. F separates entity from rephrasing for 2 models × 12 languages. Does that hold across the full panel? And why is aggregate PH ASR lower for most models: do salient named entities trigger refusal?
  • Have you re-labeled severity on a PH sample, given that labels are inherited?
  • Beyond the fixed wrappers, how do trained checkpoints hold up against adaptive attacks (PAIR/TAP), code-switching and multi-turn attacks?
  • What is the run-to-run variance of SFT/DPO? Why do Llama and Olmo get a 400k subsample?
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer HtqM from Submission21857 Authors

We thank the reviewer for their positive evaluation of our work with an exceptionally thorough and constructive review. We are glad they found the training study the most interesting part of the paper and appreciated our care for annotator welfare. Following their suggestion, we will shorten Section 5 and expand Section 6 with the new baselines and ablations. We answer all 12 questions, grouped by topic, across three parts; shared tables are in the general response.

Abbreviations: HB = HarmBench, WG = WildGuard, HB-JB = jailbreak-wrapped HarmBench, ASR = attack success rate, OR = over-refusal (%).

Q6: Can you add a template-free data baseline, a Dolci-only control and a component ablation?

Certainly, we ran all three.

(a) Template-free baseline. We translate WildGuardMix's own harmful and benign prompts into the 44 training languages, row-matched to our safety rows per language and class, with the same general data, response generator and recipe. Both reach similar ASR (1.1 vs. 1.6), but the template-free model over-refuses much more: 37.0% vs. 14.1% on HB, 39.1% vs. 7.1% on WG, 44.5% vs. 11.4% on HB-JB benign prompts, and 46.0% vs. 40.3% on the external OR-Bench. This holds with either translator (NLLB or Gemma-4) and on the 16 held-out languages (20.7 points [17.7, 23.9] lower for us). As the reviewer points out, WildGuardMix already contains WildJailbreak's contrastive adversarial-benign prompts, so slot-level minimal pairs are much more effective than prompt-level contrastive benign data. We will frame this next to WildJailbreak and Bianchi et al. Both arms are compared after SFT; we did not run DPO for the template-free arms.

(b) Dolci-only control. General data alone does not make the model safe and causes the PolyTox regression the reviewer pointed out (4.4 to 16.6), along with most of the GSM8K loss. MSafeTemp alone reaches HB ASR 2.3 with PolyTox 2.5 and GSM8K 75.5, but raises OR-Bench to 41.5, which DPO brings to 20.6.

(c) Component ablation. Harmful only gives 59.4% HB over-refusal; adding the matched benign pairs brings it to 13.1%. The localized and wrapped rows further lower over-refusal on wrapped prompts (20.5 to 16.8 to 11.0) and on OR-Bench (42.7 to 35.7). We did not ablate the BOUNDED rows separately.

Q7: Can you compare against at least one multilingual alignment method or RefusEU?

We trained MPO and RefusEU with the authors' official code, data and published hyperparameters, and evaluated them on our suite:

ModelHB ASRHB ORWG ASRWG OROR-BenchMMLU
Llama-3.1-8B-IT16.24.710.97.54.571.3
+ MPO16.94.210.78.44.971.8
+ RefusEU0.082.50.089.4100.045.0
Qwen2.5-7B-IT11.55.99.78.018.474.2
+ MPO6.17.85.512.226.773.9

For reference, our Llama SFT + DPO model (paper run): 2.0 / 5.3 / 0.8 / 2.6 / 24.8 / 63.6.

In our setup, MPO lowers ASR on Qwen2.5-7B at a moderate over-refusal cost but leaves Llama-3.1-8B unchanged, and RefusEU reaches 0% ASR with high over-refusal. As these are our retrainings evaluated under our protocol, we will report them as clearly labelled reference rows.

Trade-offs, external benchmarks and framing

  • Trade-offs. Table 3 will show all families, and the Llama and Olmo costs will be stated in the text (general response, Q2).
  • External benchmarks. For Llama, XSTest over-refusal falls from 4.4 to 0.0 after training, while JailbreakBench benign refusals rise from 5.0 to 18.4; MultiJail ASR falls from 3.3 to 0.9 (Olmo: 7.2 to 2.8). We will report all of them.
  • OR-Bench. 276 of 1,319 OR-Bench-Hard prompts (20.9%) are judged harmful by at least two of three judges. Removing them lowers over-refusal by 2.0 to 8.6 points for every trained model and keeps the order; the increase after DPO for Apertus and Olmo remains, and we will state it.
  • RQ1 and abstract. We agree that the English-to-multilingual safety gap is known, and will reframe RQ1 as a confirmation at larger scale, stressing that English results do not predict multilingual behavior in either direction. The abstract will state 44 training and 60 evaluation languages, replace "higher diversity and quality of alignment" with the result of Q6(a), and scope the localization claim as in Q9.
AuthorsOpenReview ↗

Official Response to Reviewer HtqM from Submission21857 Authors (2/3)

Q11: How do trained checkpoints hold up against adaptive attacks?

Following the reviewer's suggestion, we ran PAIR and TAP (HarmBench implementations and defaults: Mixtral attacker, 20 streams x 3 steps) on 200 HarmBench behaviors in English, German, Swahili and Hindi, against Llama-3.1-8B-Instruct and our two trained checkpoints. Since Mixtral writes weak prompts in low-resource languages, we added a second attacker (Muse-Glimmer-30B, chosen in advance by its rate of parseable prompts per language) for 18 more languages, including the 16 held-out ones.

Paired difference vs. IT, ASR in points [95% CI]MSafeTemp SFTMSafeTemp SFT + DPO
PAIR, EN/DE/SW/HI+4 [-1, +8]-9 [-13, -5]
TAP, EN/DE/SW/HI+3 [-1, +8]-9 [-13, -4]
PAIR, success only if the attack stays in the target language-12 [-16, -8]-9 [-13, -4]
PAIR, multilingual attacker, 16 held-out languages-11 [-13, -8]-15 [-18, -13]

The final model is consistently harder to attack than the released one, and so is the SFT-only model once we count only attacks that stay in the target language (against it, Mixtral often switches to English: 120 of 200 Swahili and 132 of 200 Hindi test cases). Adaptive attackers still succeed often in English (44 to 62% with Mixtral), so we describe this as a relative gain, not absolute robustness. Code-switching and multi-turn attacks remain future work.

Every wrapper strategy in the wrapped evaluation also occurs in training, which we will state explicitly. On the 41% of rows whose wrapper text never occurs in training, trained checkpoints reach 0.84% ASR vs. 16.54% for untrained ones. We will use the wording "robust to trained wrapper strategies with new payloads and new language versions".

Q8: Can you add Llama 4, Kimi, GLM, Mistral, MiniMax, DeepSeek and a closed model?

We added five large open models, evaluated with the same protocol on 25% of the seed families (cluster-bootstrap intervals in the appendix):

ModelHB TRHB PHHB ORWG TRWG ORHB-JB ASRHB-JB OR
Llama-4-Maverick5.13.516.44.826.130.210.3
Qwen3.5-397B1.10.112.20.431.00.033.0
DeepSeek-V4-Pro3.62.811.02.444.72.543.8
GLM-5.3 (thinking)1.60.42.20.420.70.112.3
Kimi-K36.15.21.85.39.410.410.7

No model is low on both ASR and over-refusal, except GLM-5.3 with thinking (which truncates 6% of replies). We will rename "frontier models" to "leading open models". We did not evaluate closed models, since sending harmful prompts at this scale conflicts with provider terms, and Mistral and MiniMax did not fit our compute budget in time (we will commit to including them in the next revisions of the submission).

Q9: Does App. F hold across the full panel, and why is PH ASR lower?

We extended the analysis to the 7 models of Table 2 plus Apertus-v1.5-8B in 58 languages, with a generic-vs-generic control (details in our response to 5PTp, Q3). With the wording fixed, the local entity behaves like a second generic sample for Apertus-8B (0.481 vs. 0.455) and differs modestly for Olmo-3.1-32B (0.473 vs. 0.398). For the localized arm of Table 2, the plain overlap is lower for 6 of 8 models, and a rate-robust measure keeps the difference significant for 3; we will revise the claim accordingly. Why localized prompts are refused more often remains open; one hypothesis is that salient named entities trigger refusal, which we will state as untested.

Q10: Have you re-rated severity on a PH sample?

Yes. On 2,000 localized/generic pairs matched by seed, language and template, with a rubric fixed in advance, localized variants are not more severe: +0.06 [0.00, +0.12] on a 1-to-5 scale with gpt-oss-safeguard-120b, and +0.04 with gpt-oss-120b (kappa 0.36 between the two).

Generator overlap. Rebuilding 50 seeds in 11 languages with Llama-3.3-70B leaves Gemma's ASR advantage unchanged (+0.6 [-1.0, +2.1]) and reduces its over-refusal advantage (+4.6 [+0.2, +9.6]); we will add a note to Table 2.

AuthorsOpenReview ↗

Official Response to Reviewer HtqM from Submission21857 Authors (3/3)

Q1: Why does Zulu have no cultural card?

The main reason is data scarcity for Zulu-specific cultural content. We will add a Zulu card in the next revision and release the corresponding data. We will also note that the pools of the HarmBench-only languages were regenerated after the run, which changes localized ASR by at most 0.25 points.

Q2: Can you evaluate the translation step directly?

This is a great question and we are happy to provide more details. MetricX-24 quality estimation on all 337,876 template pairs in 59 languages gives 4.54 on average (median 4.28; lower is better), against 15.3 for a mismatched control. Our focus is on European languages, which include several lower-resource ones; the weakest are Latvian (10.2) and Estonian (9.1), and the best overall are Urdu and German (3.3 to 3.7). Two languages outside the metric's coverage were checked by back-translation (Croatian 3.51, Swiss German 5.66; German control 2.87). MetricX-24 is trained on WMT pairs and is known to be volatile for low-resource languages such as Hausa, Yoruba, Zulu and Amharic, so we do not draw conclusions from its absolute values there. Since it scores templates, it complements but does not replace a native check of harm preservation after filling.

Q3: Who manually verified the cultural cards, for which languages, and what share was edited?

We thank the reviewer for asking this directly. The cards are generated by Gemma-4 and checked automatically. During development of the generation prompt, we reviewed and edited the cards for languages by hand; we did not record the share of edited entries. Our description in the submission was therefore too strong, and we will correct it everywhere it appears (contribution 1, Section 3.2, App. A.1 and A.3, and the AI-use statement), describing cultural authenticity as automatically checked, with human cultural-fit ratings for 11 languages (4.46/5).

Q4: Will you extend the human study and add rater demographics?

We could not extend it within the rebuttal period. Following the guidelines of our IRB-approved study, the protocol collects no personal data by design, so we can report eligibility (AI safety, red-teaming or trust and safety experience, and language competence) but not demographics. We will list the missing regions in the limitations.

Q5: How well does the judge agree with humans per language, and what share of frontier-model responses is degenerate in low-resource languages?

We did not obtain human labels for responses in the rebuttal period as finding native speakers of large number of languages (including low-resource ones) is challenging in short period of time. Applying our degenerate-response filter to the whole Table 2 panel, valid-reply shares in the low-resource tier range from 50.7% (Apertus-70B) to 99.2% (Gemma 4), and no ASR cell changes by more than 1.0 point (table in our response to 5PTp). Among valid replies in Hausa, Yoruba, Zulu and Amharic, 87 to 99% are in the target language. We will report valid-reply shares per language and training stage, and scope low-resource claims accordingly.

Q12: What is the run-to-run variance, and why the 400k subsample for Llama and Olmo?

We retrained seven checkpoints with the original recipe and seed. SFT reproduces closely (Llama train loss 0.906 vs. 0.906; cells within about 1 point, except OR-Bench +4.6). DPO varies more because its pairs are sampled on-policy (e.g., Llama OR-Bench 24.8 vs. 18.3), so we will report both values; this is a reproducibility check, not seed variance. We used the 400k subsample to test the data on further model families at a manageable cost, and we believe it is substantial enough to show the downstream effect. It keeps the Apertus blend proportions (73.5% general, 26.5% safety) and, without sequence packing, a similar number of optimizer steps (3,125 vs. 3,543). We will mark cross-family comparisons accordingly.

Benign labels. We adjudicated all 94 benign judgments that raters disputed: none was judged harmful, 31 were judged borderline, and over-refusal on human-endorsed items closely tracks over-refusal on all items (details in our response to q5qT). We will mark the borderline items in the release.

Further revisions. A language inventory table (family, resource tier, train/eval membership, coverage gaps), and corrected references (Gemini, outdated venues, missing citations), together with the proposed works, including SEA-Guard and XL-SafetyBench.

We are grateful for the reviewer's detailed list, which substantially improved various sections of our submission, and we are happy to discuss any of the points further.

AuthorsOpenReview ↗

Follow-up from Submission21857 Authors

Dear Reviewer HtqM,

Thank you again for your thorough review and overall positive evaluation of our submission. Following your suggestions, we have uploaded the revised PDF (changes in blue) which shortens Section 5 and expands Section 6. In particular:

  • Q6: template-free baseline, Dolci-only control and component ablation (Table 5; App. L.1, L.2)
  • Q7: MPO and RefusEU as reference rows (Table 29)
  • Trade-offs: all families in Table 3, and capability by starting checkpoint in Table 4
  • Q11: wrapper overlap (App. J.1) and PAIR/TAP (App. J.2, Table 55)
  • Q8: five large open models (App. D.3, Table 31)
  • Q9 and Q10: generic-vs-generic control and severity re-rating (App. F.1, F.2), with the localization claim revised in Section 5
  • Q1 to Q4: language inventory (Table 10), translation quality (App. M.7), the corrected cultural-card description (App. A.1, A.3 and the AI-use statement), and the human-study limitations in Section 7
  • Q5: validity gate (App. D.4) and valid-reply shares per language and training stage (App. E.3)
  • Q12: reproducibility check (App. L.3, Table 70) and the 400k subsample (App. L)
  • External benchmarks and the OR-Bench audit: Table 30 and App. D.2
  • References: corrected and extended in Section 2
  • Linguistic gap: expanded in Limitations section referencing points from review's weaknesses section.

As the discussion period ends soon, we wanted to check whether our response and the revision address your concerns, and we are happy to discuss any of the points further.

Best regards,

Authors

Reviewer_cX4AOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MSAFE-TEMP, a template-based approach for scaling multilingual safety alignment data. The method extracts harmful elements from existing prompts, translates the resulting templates, and fills them with harmful, benign, or culturally relevant entities. Using this pipeline, the authors construct a large dataset covering 44 languages for training and 60 languages for evaluation. They evaluate several open-weight models to examine multilingual safety gaps, particularly in low-resource languages, and assess the impact of culturally localized prompts on attack success and over-refusal. The authors further validate the dataset through automated judging and a small human evaluation study. Fine-tuning experiments using SFT and DPO show improvements in multilingual safety, including generalization to languages not seen during training, although some improvements come at the cost of reduced model capabilities.

Critical strengths

I identify two main strengths of this work. The first is the large-scale multilingual safety dataset, and the second is the extensive evaluation and evidence of cross-lingual generalization. I provide further details below.

  1. Large-scale multilingual safety dataset. One strength of this work is the scale and coverage of MSAFE-TEMP. The authors construct approximately 1.8 million validated prompts across 44 languages for training, alongside a separate evaluation benchmark covering 60 languages. I particularly appreciate the inclusion of culturally localized harmful prompts and matched benign examples, which allows the authors to evaluate both unsafe compliance and over-refusal. The dataset and generation pipeline could be useful resources for future multilingual safety research.

  2. Extensive evaluation and cross-lingual generalization. The authors evaluate several open-weight models across different languages and show that culturally localized prompts can reveal safety failures missed by direct translation. I also find the results in Appendix E.1 interesting, where safety fine-tuning improves performance on languages excluded from training. This provides useful evidence of cross-lingual transfer beyond the languages represented in the training data.

Critical weaknesses
  1. Limited evidence for the benefit of cultural localization. In Section 5, the authors report that culturally localized prompts reveal different safety failures from translated prompts. However, it is unclear whether these additional failures result from cultural localization or simply from changing the entities in the prompts. A comparison with independently generated generic variants would help establish whether cultural localization offers a distinct advantage over ordinary prompt variation.

  2. Significant capability degradation after safety training. Although the proposed training improves multilingual safety, it also leads to substantial performance drops on general benchmarks. For instance, Table 24 shows that Llama-3.1-8B drops from 84.0 to 68.5 on GSM8K and from 72.0 to 63.6 on MMLU after SFT and DPO. Over-refusal on OR-Bench also increases from 4.6 to 24.8. These trade-offs raise concerns about whether the safety improvements justify the loss in general capabilities and helpfulness.

Questions for authors
  1. In Section 5 and Appendix F, the authors report that culturally localized prompts identify safety failures missed by generic translated prompts. How can we determine whether these failures are specifically due to cultural localization rather than ordinary variation in the entities used to fill the templates? Have the authors considered a control experiment using independently sampled generic variants?

  2. Table 24 shows substantial capability degradation following safety fine-tuning. For example, Llama-3.1-8B drops from 84.0 to 68.5 on GSM8K and from 72.0 to 63.6 on MMLU after SFT and DPO. Could the authors clarify why these losses are relatively large and whether the safety improvements can be achieved without such significant reductions in general capabilities?

4 comments
AuthorsOpenReview ↗

Official Response to Reviewer cX4A from Submission21857 Authors

We thank the reviewer for their overall positive assessment, constructive review and for highlighting the scale of the dataset, the matched benign examples and the transfer to held-out languages. Both questions led us to new experiments that make the paper's claims more precise, and we address them below.

Before that, we would like to point the reviewer to the template-free baseline in the general response, the result that most changed our own view of the paper. At the same data budget and the same multilingual safety, translated WildGuardMix leads to 23 to 32 points more over-refusal than MSafeTemp, also on the 16 held-out languages. We believe this makes the case for the template design more directly than the submission did, and we will make it the central result of Section 6.

Q1: Are the failures found by localized prompts specific to cultural localization, or do they come from ordinary variation in the entities?

We ran the control the reviewer proposed. For each template, we drew two disjoint sets of generic fills, each matched in size to the localized arm, and compared the overlap of broken behaviors (Jaccard) for generic vs. generic and for generic vs. localized, over 500 resamples with cluster-bootstrap intervals over behaviors.

With the wording held fixed (the ablation of App. F, 12 languages), only the entity changes:

ModelGeneric vs. genericGeneric vs. localizedDifference [95% CI]
Apertus-8B0.4810.4550.026 [-0.028, 0.081]
Olmo-3.1-32B0.4730.3980.075 [0.029, 0.121]

Here the reviewer's intuition is confirmed: the local entity on its own reaches about the same new failures as a second generic sample (Apertus) or differs only modestly (Olmo). We will therefore replace the "about 45% of arm-specific failures occur only with local entities" statistic with this control.

For the localized arm as a whole, the entity comes with its own local framing, since a request about a local context is naturally phrased differently. We extended the analysis to the 7 models of Table 2 plus Apertus-v1.5-8B, in 58 languages. The plain overlap is lower for 6 of 8 models (pooled 0.388 vs. 0.293). Localized prompts also break fewer behaviors for most models, which lowers Jaccard by itself, so we lead with a rate-robust measure (does the generic arm also break the cells that the localized arm breaks). Under it, the difference remains significant for 3 models: Gemma 4, Olmo-3.1-32B and Apertus-8B.

The precise claim we will make in the abstract, Section 5 and App. F is therefore: localized prompts reach failure sets that generic resampling does not reach for some models, and the entity alone does not explain this.

Beyond evaluation, the localized data also helps training. In our component ablation (general response), adding the localized rows lowers over-refusal on jailbreak-wrapped benign prompts from 20.5% to 16.8% and on OR-Bench from 42.7% to 36.7%, at similar ASR (1.9 vs. 2.2). Localized variants are also not more severe than generic ones (+0.06 [0.00, +0.12] on a 1-to-5 scale over 2,000 matched pairs). We agree with the reviewer that the main value of the arm is coverage of locally relevant content, which generic or translated data does not contain, and we will present its contribution in these terms.

AuthorsOpenReview ↗

Official Response to Reviewer cX4A from Submission21857 Authors (2/2)

Q2: Why are the capability losses large, and can the safety gains be achieved without them?

The reviewer is right that the cost for Llama is substantial, and we will state it directly in the main text. Following the question, we looked at where the cost comes from and how it depends on where MSafeTemp enters training.

Llama-3.1-8B (paper run)MMLUGSM8KPolyToxOR-Bench
Released IT model72.084.04.44.6
+ general data only65.469.716.618.3
+ MSafeTemp only65.175.52.541.5
+ both (SFT)65.267.96.735.7
+ both (SFT + DPO)63.668.56.124.8

Fully fine-tuning an already instruction-tuned model costs about 7 MMLU points with any data, including general instruction data alone, which also accounts for most of the GSM8K loss and the toxicity increase. This can be attributed to the fact, that our SFT and DPO training mixes do not match the scale and quality of original Llama 3.1 8B alignment, therefore, capability loss is expected. Training on base checkpoints across models shows lower capability losses with addition of more safety data, as explained further below.

Integrated into post-training, the cost is much smaller. A released instruct model is the hardest place to add data. When MSafeTemp is added on top of the vendor SFT checkpoint and followed by DPO, as a developer would add it to their post-training mix, capability is largely preserved (Tables 24 and 25):

Vendor SFT, then + MSafeTemp SFT + DPOWG ASRMMLUGSM8KIFEval
Apertus-8B11.0 → 0.558.1 → 59.856.8 → 52.961.0 → 71.0
Olmo-3-7B5.6 → 0.562.9 → 61.777.1 → 74.676.5 → 77.3

This is the setting we intend MSafeTemp for. Lighter adaptation of finished models (adapters, lower learning rates) remains open, and we list it as future work.

Over-refusal. We agree that the gap between in-distribution over-refusal after DPO (5.3%) and OR-Bench (24.8%) deserves a prominent place, and we will discuss it in the main text. Three observations suggest it is not template-style overfitting:

  • On the hand-written XSTest, over-refusal of our final Llama model falls from 4.4% to 0.0% (JailbreakBench benign refusals rise from 5.0% to 18.4%, which we also report).
  • At the same data budget, template-free data raises OR-Bench more than MSafeTemp (46.0 and 48.7 vs. 40.3 after SFT), so the increase is shared by multilingual refusal data in general and is smallest with templates.
  • After MSafeTemp, OR-Bench lands between 25% and 50% across families. It falls for checkpoints that over-refuse strongly out of the box (Apertus IT 82.6 → 34.6, vendor SFT 81.7 → 36.9) and rises for those that rarely refuse (Llama IT 4.6 → 24.8). For reference, the released Qwen3.5 model scores 79.8.

In addition, about 21% of OR-Bench-Hard prompts are judged harmful by at least two of three judges; removing them lowers our models' over-refusal by 2.0 to 8.6 points.

Safety gains. Multilingual ASR falls from 15.9% to 2.0% on HarmBench and from 10.5% to 0.8% on WildGuard, with over-refusal on matched benign pairs after DPO (5.3% and 2.6%) at or below the released model (4.5% and 7.4%). The transfer to held-out languages also extends to adaptive attacks: with PAIR in the 16 held-out languages, our final model is attacked successfully 15 points less often (-15 [-18, -13]).

We hope that our responses address the reviewer's concerns, and we remain happy to answer further questions.

Thank you for the additional experiments. They address my concerns. I find this study very valuable and important to multilingual AI. Kindly add the new findings to the appendix.

AuthorsOpenReview ↗

Follow-up from Submission21857 Authors

Dear Reviewer cX4A,

Thank you again for your valuable feedback, kind words and overall positive assessment of our work. We have updated the submission revision and the new PDF version (changes marked in blue) reflects both of your questions:

  • Localization vs. entity variation (Q1): the generic-vs-generic control you proposed is in App. F.1 (Table 45), and the severity re-rating in App. F.2 (Table 46). Section 5 now states the narrower claim, and presents the localized arm as widening coverage of locally relevant content rather than as a stronger attack.
  • Capability cost (Q2): Table 4 in the main text shows the capability change by starting checkpoint. The cost is largest on top of released instruct models and small at the vendor-SFT stage. Table 5 separates the general data from MSafeTemp. Section 6 discusses the gap between in-distribution over-refusal and OR-Bench, and Section 7 lists lighter adaptation as future work.
  • Template-free baseline: now the central comparison of Section 6 (Table 5).

Table numbers in our response refer to the submission (Tables 24 and 25 are now Tables 27 and 28). We remain happy to answer further questions if there are any.

Best regards,

Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Official General Response from Submission21857 Authors

We thank all reviewers for their time, their detailed feedback and the many concrete and valuable suggestions. We are happy to hear that reviewers found that our work addresses an important and under-studied problem of multilingual safety alignment (q5qT, HtqM), contributes training data and training experiments rather than only another benchmark (HtqM), provides a large and carefully engineered resource with matched harmful, benign and localized variants (cX4A, 5PTp, q5qT), and is supported by a broad and transparent evaluation (cX4A, 5PTp, HtqM).

During the rebuttal period we ran the controls the reviewers asked for. Below we summarize the experiments shared across reviews, and we address each reviewer's remaining questions individually. We will include all new results in the revised version.

Abbreviations: HB = HarmBench track, WG = WildGuard track, HB-JB = jailbreak-wrapped HarmBench, ASR = attack success rate, OR = over-refusal on benign prompts. All values in %; lower is better except MMLU and GSM8K.

Q1: Do the gains come from the template design or simply from having more multilingual safety data? (q5qT, 5PTp, HtqM)

We trained Llama-3.1-8B-Instruct with the same recipe, the same general data and the same safety-data budget (row-matched per language and class, about 106k safety rows), changing only the source of the safety prompts. As the template-free baseline, we take WildGuardMix's own harmful and benign prompts (47,851 unique prompts after deduplication and removing any overlap with our test seeds, HarmBench and OR-Bench) and translate them into the 44 training languages, either with NLLB-200 3.3B or with Gemma-4-31B. Answers for all arms come from the same generator and judge filter, so only the prompts differ.

Safety dataHB ASRHB ORWG ASRWG ORHB-JB OROR-BenchMMLU
None (released model)16.24.710.97.58.04.571.3
WildGuardMix, NLLB1.137.01.139.144.546.064.0
WildGuardMix, Gemma-41.137.70.938.247.748.765.4
MSafeTemp1.614.10.77.111.440.364.7

All rows: SFT, one run each, same evaluation stack (values can differ by up to about 1 point from Table 3 of the submission).

Both data sources reach similar multilingual safety, but the template-free models refuse benign requests far more often. Paired per language, MSafeTemp lowers over-refusal by 23.1 points on HarmBench [21.1, 25.3] and 32.5 points on WildGuard [28.4, 36.5], also on the 16 languages never seen in training (20.7 points [17.7, 23.9]). On the external OR-Bench, which neither pipeline produced, the gap is smaller but in the same direction (40.3 vs. 46.0 and 48.7). Our ASR is 0.6 points higher [0.4, 0.8]. The translator makes little difference (Gemma-4 minus NLLB on HB OR: +0.7 [-0.2, +1.9]).

Our component ablation shows where the difference comes from (based on Llama-3.1-8B-Instruct training runs):

Safety data (cumulative)HB ASRHB ORWG ORHB-JB OROR-Bench
Harmful only0.659.471.158.857.5
+ matched benign pairs1.913.16.920.542.7
+ localized2.211.57.116.836.7
+ jailbreak-wrapped (full MSafeTemp, paper run)1.714.16.811.035.7

Notably, WildGuardMix already contains contrastive benign prompts, including the adversarial benign prompts of WildJailbreak, yet the template-free models still over-refuse at 37%. We believe the key is that our benign examples are minimal pairs of the harmful ones: same frame, same language with the slots being the only change. This is what the template design makes possible at scale both horizontally (size) and vertically (language coverage), and we will make it the central result of Section 6.

AuthorsOpenReview ↗

Official General Response from Submission21857 Authors (2/2)

Q2: Where does the capability cost come from? (cX4A, HtqM, 5PTp)

We separated the general instruction data from the safety data:

Training data (Llama-3.1-8B-IT)HB ASRPolyToxOR-BenchMMLUGSM8K
None (released model)15.94.44.672.084.0
General data only(*)16.618.365.469.7
MSafeTemp only2.32.541.565.175.5
General + MSafeTemp (SFT)1.76.735.765.267.9
General + MSafeTemp (SFT + DPO)2.06.124.863.668.5

(*) Only 37 to 58% of this model's replies are scorable, so we do not report its ASR.

Fully fine-tuning the released instruction-tuned model costs about 7 MMLU points with any of these data sources. This part of the cost comes from our full fine-tuning recipe rather than from the safety data, and we will state it clearly. The general data accounts for the toxicity increase the reviewers noticed (PolyTox 4.4 to 16.6) and for most of the GSM8K loss, while MSafeTemp on its own brings the safety gain with lower toxicity than the released model. MSafeTemp alone still costs 8.5 GSM8K points and raises OR-Bench, which DPO lowers again (41.5 to 20.6). We will replace the description of the capability cost in the main text with these numbers for all families, and discuss lighter adaptation of instruction-tuned models (adapters, lower learning rates) as future work.

Q3: How reliable are the benign labels? (q5qT, HtqM, 5PTp)

In our expert study, raters endorsed 65.3% of the 300 benign judgments. The disputed ones are borderline requests: raters were either unsure about the label or noted that the request may be harmful in particular, rare contexts. Two human raters independently adjudicated all 94 disputed judgments as benign, borderline or harmful:

Benign judgments (n = 300)Share
Endorsed by raters65.3%
Endorsed or adjudicated benign85.7%
Adjudicated borderline10.3%
Adjudicated harmful0%

Over-refusal on human-endorsed items closely tracks over-refusal on all items, so the disputed items do not drive our results. We will mark the borderline items in the release. Details are in our response to q5qT.

Summary of new experiments

  • Template-free translation baseline with two translators, and a data-component ablation (q5qT, 5PTp, HtqM)
  • Decomposition of general vs. safety data (cX4A, HtqM, 5PTp)
  • Generic-vs-generic Jaccard control with bootstrap intervals, on the 7 models of Table 2 plus Apertus-v1.5-8B in 58 languages (cX4A, 5PTp, HtqM)
  • Adaptive attacks (PAIR and TAP) against the trained models in 22 languages (HtqM)
  • Five additional large open models: Llama-4-Maverick, Qwen3.5-397B, DeepSeek-V4-Pro, GLM-5.3 and Kimi-K3 (HtqM)
  • Degenerate-response filter applied to the full Table 2 panel (5PTp, HtqM)
  • Human adjudication of all 94 disputed benign judgments, with over-refusal on human-endorsed items (q5qT, HtqM, 5PTp)
  • Safety-judge check of all 408,816 training answers against their response class (5PTp, q5qT)
  • Translation-quality estimation on 337,876 template pairs, severity re-rating of localized variants, a generator self-bias check, an OR-Bench audit and external benchmarks (HtqM, 5PTp, q5qT)

Revisions we will make

  • Abstract: state 44 training and 60 evaluation languages, and scope the localization claim to what the new control supports.
  • Section 6 becomes the central experimental section with the baselines, ablation and capability decomposition; Section 5 is shortened.
  • Trade-offs for all model families move into the main text.
  • A language inventory table (family, resource tier, train/eval), a notation block, and valid-reply shares per language and training stage.
  • A precise description of how the cultural cards were checked (details in our response to HtqM).
  • Corrected and extended references, including the related work suggested by reviewers.

We plan to update the submission PDF in the upcoming days, while initially providing new findings in the respective discussion threads. We hope these results address the reviewers' main concerns, and we look forward to the discussion.

AuthorsOpenReview ↗

Rebuttal Summary from Submission21857 Authors

Rebuttal summary

We thank all reviewers for their time and feedback, and the Area Chair for managing the discussion. Reviewers agreed that our work addresses an important and under-studied problem (q5qT, HtqM), contributes a large and carefully engineered resource with matched harmful, benign and localized variants (cX4A, 5PTp, q5qT), and is supported by a broad and transparent evaluation (cX4A, 5PTp, HtqM). After the rebuttal, Reviewer cX4A confirmed that the new experiments address their concerns and described the study as "very valuable and important to multilingual AI", and Reviewer 5PTp confirmed that they address most of their concerns.

The main concerns were addressed with new experiments, all included in the revised PDF (changes in blue):

  • Templates vs. more multilingual data (q5qT, 5PTp, HtqM): at the same data budget and near-identical safety (ASR within 0.6 points), translated WildGuardMix over-refuses 23 to 32 points more than MSafeTemp on our benign pairs, also on 16 unseen languages and with either translator. The component ablation attributes the gain to the matched benign pairs (Section 6, Table 5; App. L).
  • Reliability of benign labels and data quality (q5qT, 5PTp, HtqM): all disputed benign judgments were adjudicated (85.7% endorsed or adjudicated benign, 10.3% borderline, none harmful), and the disputed items do not drive over-refusal. All 408,816 training answers were checked against their response class (0.35% of the training mix violates it), and translation quality was estimated for all languages (App. M.5 to M.7). Our human study requires raters who are native or fluent speakers with experience in AI safety, red-teaming or trust and safety, under an IRB-approved protocol that limits exposure to harmful content. Such raters are scarce for most of our 60 languages and the protocol rules out crowd annotation, so we complement the 11-language expert study with automatic checks in all languages.
  • Localization claim (cX4A, 5PTp, HtqM): a generic-vs-generic control and a severity re-rating; the control supports the claim for a subset of models, as stated in Section 5 (App. F.1, F.2).
  • Capability cost (cX4A, 5PTp, HtqM): the cost is largest on top of released instruct models and smaller at the vendor-SFT stage, where the data is intended to be used (Table 4).
  • Baselines and robustness (HtqM): MPO and RefusEU reference rows, PAIR and TAP adaptive attacks, five large open models, external benchmarks, and a reproducibility check (Tables 29 to 31, 55, 70).

Remaining limitations, stated in Section 7 and the appendix: the human study covers 11 languages with one or two raters each; the response judge is not validated on low-resource responses; training configurations are single runs (with a reproducibility check in App. L.3); and on the external OR-Bench, the over-refusal gap points the same way but is close to run-to-run variation.

We thank the reviewers again for their suggestions, which substantially improved the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.