acceptodds
Desk rejected at ICLR 2027

Safety Alignment Under Language Shift: Harmful Compliance, Over-Refusal, and Response-Language Dynamics in Low-Resource Languages

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model safety alignment is usually evaluated in high-resource lan- guages, leaving low-resource settings underexplored. We evaluate proprietary and open-source models across English, Mandarin Chinese, Cantonese, and Ti- betan using benign and borderline prompts for false refusal and AdvBench for harmful compliance. We additionally measure off-topic output, repetition, and target-language response rate, and manually verify ASR-relevant labels. For ev- ery proprietary model with paired data, harmful compliance is higher in Cantonese and Tibetan than in its English–Mandarin baseline, whereas false refusal changes heterogeneously. Under unconstrained harmful prompting, Cantonese response- language consistency decreases substantially for several models, showing that aggregate safety metrics and response-language behavior should be interpreted jointly. Explicit target-language instructions largely restore language consistency but do not consistently improve safety. Among open-source models, low ASR can reflect off-topic generation or repetition, with off-topic behavior occurring across both benign and harmful datasets. Multilingual evaluation should jointly distinguish harmful compliance, false refusal, relevance, validity, and response- language behavior.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multilingual Safety Alignment, Low-Resource Languages, Harmful Compliance, Over-Refusal, Response-Language Consistency, Jailbreak Robustness, Failure Mode Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5akVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies safety in English, Mandarin, Cantonese and Tibetan. They carefully study safety in terms of harmful request refusals, over-refusal, and errors arising from other reasons (such as a lack of capabilities).

Critical strengths

The paper identifies and studies pressing issues. The combined study of safety and multilingual evaluation is important and often-times neglected. They also correctly motivate, in my opinion, that harmful compliance can arise from diverse sources: poor instruction following, irrelevant generation, repetition, etc. This is important work and I would like to see more such studies accepted and published.

Critical weaknesses

I have 2 major issues with this work. The first concerns limited evaluation and over-claiming. The title and framing references low resource languages, but there's no evidence presented to show how Tibetan and Cantonese fit this description. Why focus on these two languages in particular? Why exclude other low-resource languages? Why do results from these languages allow us to make claims about low resource languages in general? Either the framing should be changed to align with the actual experiments, or the experiments should be suitably expanded to match the framing.

The second concerns specific issues with the ambiguous descriptions and (lack of) detail in the paper. These might seem like cosmetic issues on the surface, but since the main thrust of this work seems to be to shed light on an under-represented issue and provide a good evaluation benchmark, presentation clarity is key. I detail a subset of my concerns in "Questions For Authors".

Questions for authors
  1. figure 1 reads like a poster for the paper, and adds limited value. Can this perhaps be replaced with an example illustrating the phenomenons being studied? A single datapoint selected from the evaluations to illustrate the main issues highlighted in lines 44-50?

  2. The term "ASR" is used before being defined.

  3. line 76: what is a "matched model-language" pair?

  4. line 137: why is the Tibetan result unavailable?

  5. line 167-168: if these are not counted as harmful compliance, are they counted as refusal or as off-topic? Why should such a detail be omitted?

  6. lines 170-172: the FRR definition is circular. The same paragraph completely omits the definitions of off-topic and substantive-engagement.

  7. line 186: does adjudication mean human adjudication?

  8. line 192: which labels?

  9. lines 202 to 212: can these numbers be tabulated? Can we instead use the text to define what "human review" means (who? how? what protocol?)?

  10. figure 2 should be a table(s) instead.

  11. line 318-319: FRR is (also) a capability measure, so how can it be used to infer there is no shift in refusal tendency?

  12. lines 501-502: how was political, idealogical, and other sensitive content filtered out of the dataset? Aren't these components an integral part of safety evaluations? Will the data be open-sourced for inspection (instead of by request only, as detailed in the reproducibility statement)?

Reviewer_gNRSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a LLM safety evalution in low resource settings. Authors experiment across 4 languages (English, Mandarin, Cantonese, and Tibetan) and several close and open weight models. They curate a dataset by picking data from several dataset AdvBench, XSTest and OR-Bench. They measure harmful comliance, false refusal rate, off topic drift, repetitive behavor etc. Authors find higher harmful comliance rate for the Cantonese and Tibetan compared to English and Mandarin. They also evalaute response generation prompting models with a system prompt aksing to detect prompt language which reduces the response generation in languages other than the input language.

Critical strengths

I would like to mention the below most sutiable strengths of the paper:

  1. Good number of proprietary models evaluated (although lesser open weight models)
  • The evaluation results of the modern proprietary models are valuable, while more open weight models should be evaliated
  1. Combination of harmful compliance, false refusal rete and response-language consistency
  • The combined analysis of these metrics makes the evluation comprehensive and meaningful
  1. Human verification
  • Authors conduct LLM judge and human verification which are critical steps in LLM safety measurement work
Critical weaknesses

I would like to mention the below most important weaknesses of the paper:

  1. Limited methodological novelty, incremental evaluation methodology and mostly existing metrics from literature.
  • no new method proposed for evaluating LLM safety in low resource settings. the metrics from literature are combined for evaluating the safety. no systematic translation quality measurment except the mention of human validation
  1. Insufficient differentiation from closely related prior work and no experimental comparison with prior work
  • authors do not differentiate thier work with the work from literature, what new contribution they create in this work except the redoing of the harmful comliance, irrelevant generation, false refusal rate etc. which are present in literature
  1. Reclaiming already addressed low-resource safety gap, lower number of languages and limited generalizability
  • in literature several work have benchmarked LLM safety for low resource languages with larger number of languages
  1. The anonymous link is not accessible which prevents to check the reproducibility along with the lacking of details of the reproducibility in the paper
Questions for authors

I would suggest authors to answer the below questions and enhance the paper with findings from the answer to these questions:

  1. Can you do a baseline comparison by evaluating the work from literature () using the same set of dataset and report the findings and compare with your results and list the new findings achieved not present in literature
  2. Safety Alignment Under Language Shift - to claim this phrase can you expand the number of low resource languages (at least higher than any work from literature) to generalize the claims
  3. Can you perform some translation quality measurement and report some metrics like F1 score, COMET kiwi score etc.
  4. Can you share the codebase to check the reproducibility of the work, currently the anonymous link is not accessible

Work from literature to compare with: https://aclanthology.org/2024.findings-acl.156 https://github.com/AIM-Intelligence/XL-SafetyBench https://arxiv.org/pdf/2508.12733 https://proceedings.iclr.cc/paper_files/paper/2024/file/6b396f766a50e0853a5164e68048540c-Paper-Conference.pdf

Reviewer_rhMZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • Evaluates 11 API-served models and a set of older open models (Llama-2, Chinese-Alpaca, Qwen2.5-7B, Cantonese/Tibetan adaptations) on English, Mandarin, Cantonese and Tibetan.
  • Harmful compliance uses translated AdvBench (520 prompts). Over-refusal uses 250 XSTest + 250 OR-Bench Hard prompts per language. Off-topic, repetition and target-language response rate (TLRR) are also measured, and Cantonese/Tibetan are tested with an explicit "respond in X" instruction.
  • Claims: (i) ASR rises in Cantonese and Tibetan for every proprietary model while FRR shifts heterogeneously; (ii) low ASR in open models partly reflects off-topic output; (iii) language constraints restore TLRR but not safety.
Critical strengths
  • Good balance of metrics: ASR, textual FRR, off-topic, repetition and TLRR are reported per model–language pair (l.250–256; Tabs. 1–8).
  • Manual translation verification by native-speaker co-authors; the Cantonese over-refusal set is reviewed twice (l.141–150).
  • The human audits are reported openly, including unfavourable results: off-topic precision is 0/50 for proprietary Cantonese and 1/45 for Tibetan (l.202–212; App. A.2).
  • Tibetan has received little attention in safety work so far, and the model set is current. The over-refusal translations and code are released and make for a nice contribution(l.508–514).
Critical weaknesses

Scope, novelty, framing

  • Both "low-resource" languages are still Sino-Tibetan, as is the Mandarin baseline: 3 of the 4 languages come from one family. The paper's general claims about low-resource languages (l.015–029, l.438–466, l.471–481) are therefore not supported beyond this family.
    • Cantonese (~85M speakers, shares a script with Mandarin) is a debatable "low-resource" choice. The paper gives no data or resource measure to back the label (l.139–140).
  • The novelty claims are wrong or overstated.
    • Same model–language pairs compared across languages (l.117–119): done before [1, 3, 6, 10, 11].
    • Jailbreak robustness and over-refusal evaluated jointly (l.127): done before [3, 6, 10, 11].
    • Multilingual over-refusal data "scarce" (l.126): existing resources [3, 6, 10].
    • The claim that low ASR can come from off-topic output (l.91–93), was reported earlier [2], and three-way refused/complied/unclear (or deflection) judging is established [4, 5].
    • The FRR–ASR "no universal trade-off" argument (l.362–366) ignores the work that established the trade-off [9] and IRT-based decompositions of jailbreak success [7]. Heterogeneous over- and under-safety in low-resource languages was already reported [8].
  • The work is highly repetitive. The same three findings are restated in the abstract (l.015–029), Fig. 1, the contributions (l.085–095), results, discussion (l.438–466) and conclusion (l.471–481).
    • App. A l.638–647 copies l.340–354 verbatim. "Kimi K2.5 Tibetan unavailable" is repeated about 8 times.
  • The target-language consistency analysis seems tangential. TLRR does not move ASR or FRR (l.321–331), and the section is not connected to the safety findings.
    • TLRR is not broken down by response outcome. The Cantonese AdvBench TLRR drop (DeepSeek 15.6%, Qwen3.6 21.5%, Doubao 22.7%; Tab. 7) might simply be refusals emitted in Mandarin or Standard Written Chinese. The paper does not split TLRR by outcome.
  • The results section adds little beyond what the figures and tables already show.
    • The cross-language results (l.263–318) restate the per-language mean ASR and FRR from Fig. 2 and add two model examples.
    • Four paragraphs (l.336–366) are spent on FRR–ASR correlations. None is significant in the low-resource conditions, and the two sensitivity checks repeat the same conclusion.
    • The proprietary off-topic analysis (l.370–377, 421–423) mainly concludes that the Tibetan off-topic rates are unreliable.
    • The open-model analysis (l.427–433) is two short paragraphs on Tab. 2.
    • Missing: a per-harm-category breakdown, qualitative examples, and any analysis of causes (tokenization, refusal language, API-level filtering).

Methodology & presentation

  • The method section is badly structured.
    • XSTest and OR-Bench are introduced as part of the evaluation halfway through "Models and Datasets" (l.145), after the AdvBench details.
    • Metrics are defined only at the end of the methods, after Fig. 2 (l.250–256). Human-verification details come before the metric definitions.
    • Key setup details (model versions, decoding settings, judge prompts, translation procedure) are deferred to the supplementary material (l.151–152, l.178–179), so the methods section alone does not define the evaluation. That is a problem for a paper whose main contribution is the evaluation setup.
    • Models are not listed in the methods; they appear only in figure legends ("supplementary", l.151). Open models are first named in Tab. 2. The judge and adjudicator models are not named (l.176–178).
  • The open/closed model classification is wrong. 5 of the 11 "proprietary" models are open-weight on Hugging Face: GLM-5.1, Kimi K2.5, MiniMax-M2.7, MiMo-V2.5-Pro, and DeepSeek-Reasoner (the API name for the reasoning mode of DeepSeek's open V3.x/V4 models). Yet all are labelled "proprietary" (l.134; Figs. 1–3; Tabs. 1, 3–8). Being accessed through an API does not make a model proprietary.
    • "DeepSeek-Reasoner" is an API endpoint name, not a model ID. DeepSeek has served several models behind it over time (V3.1, V3.2, V4), so readers cannot tell which model was evaluated and the result cannot be reproduced once the endpoint moves on. Report the model ID and version.
    • As a result, "proprietary vs. open-source" is effectively a comparison of 2026 frontier models with 2023–24 models of 7–13B parameters. Model scale and age confound it, so the open-model findings (l.91–93) may not hold for current models.
  • Some evaluations cover only proprietary models: the language-constraint intervention (l.138), the off-topic table (Tab. 1) and the main FRR–ASR correlations.
    • The open-model set differs per language (Tab. 2). Only Qwen2.5-7B appears in all four languages, so there is no paired cross-language comparison for open models.
  • The Kimi K2.5 Tibetan exclusion is not justified. The main text only says the result is "unavailable" (l.136–137; Fig. 2/3 captions). App. B (l.803–805) explains that Moonshot's API "frequently triggered service-level error responses" for Tibetan input.
    • Kimi K2.5 is open-weight (moonshotai/Kimi-K2.5 on HF), so the authors could have run it locally or through another provider and obtained a Tibetan result without the API filter.
    • The API-level blocking of Tibetan input is itself worth reporting as a result, since it amounts to blanket refusal of a whole language. Excluding the model affects the Tibetan means and the "every model" claim.
  • Fig. 2 is hard to read. Model IDs 1–11 must be looked up in the footer. The colour scale is poor: with a 0–70 colormap, the differences that matter (ASR 1–15%, FRR 10–25%) barely show up. The proprietary label is wrong here too.
    • Fig. 3 has no colour→model legend at all, and its y-ranges differ across panels. Two outliers (DeepSeek FRR 62.6%, Doubao ASR 60.6%) drive the Tibetan correlations.

Validity of headline numbers

  • Labels were corrected for Cantonese only, which inflates the cross-language gap.
    • All Cantonese ASR labels were reviewed by hand (l.192–194), and 27/50 proprietary "off-topic" responses turned out to be harmful (l.204–205).
    • The paper reports no review of Mandarin labels, although the classifier marks 15–21% of Mandarin responses as off-topic for Claude, GLM, Gemini and MiniMax (Tab. 3). At the Cantonese relabel rate, Claude and Gemini would have higher ASR in Mandarin than in Cantonese, and the mean gap would shrink from ~8 pp to ~2–3 pp. The claim that "every model" has higher ASR (l.88, l.268) may not hold.
    • Tibetan was not corrected either, although 17/45 Tibetan off-topic responses were harmful (l.205–206).
  • Empty and error outputs count as refusals for ASR (l.165) but not as false refusals for FRR (l.172). This affects large shares of responses: Claude 21.5%/24.4% empty on Cantonese/Tibetan AdvBench, and Qwen3.6 46.4% errors on constrained Tibetan (Tab. 7). API-level blocking therefore lowers ASR without raising FRR.
  • The statistics do not support the FRR–ASR conclusions. The correlations use n = 10–11 models, with no confidence intervals. "Not stable across languages" (l.350–351) is inferred from differing p-values; the differences between the r values are not tested.

Citations / related work

  • [1] Friedrich et al., LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Gaps (arXiv 2412.15035): same models and prompts across languages; per-category gaps even among high-resource languages, which challenges pooling English and Mandarin as one baseline (l.140).
  • [2] Shen et al., The Language Barrier (ACL Findings 2024): more irrelevant responses to malicious prompts in low-resource languages, which anticipates the off-topic finding (l.91–93).
  • [3] Aziz, Hanif & Koto, Low-Resource Safety Failures Are Action Failures (2606.01196): joint harmful/harmless refusal across 23 languages, with the same ASR↑/FRR-not-uniform finding.
  • [4] Dahir, SomaliBench Eval (2605.25420): the main non-refusal mode is "unclear" output (wrong-language, off-topic), judged three-way with a native-speaker check.
  • [5] Akinode et al., TukaBench (2606.01322): a "Deflection" outcome, plus evidence that LLM-judge agreement with humans drops in low-resource scripts. This should motivate the audits and supports the translation caveat.
  • [6] Ning et al., LinguaSafe (2508.12733): multilingual harmfulness and oversensitivity evaluated jointly.
  • [7] Zhang et al., Why Do Safety Guardrails Degrade Across Languages? (2605.17173): IRT decomposition showing that aggregate jailbreak success confounds several factors.
  • [8] Pattnayak & Chowdhuri, IndicSafe (2603.17915): heterogeneous over- and under-safety in 12 low-resource languages.
  • [9] An et al., PHTest (2409.00598): establishes the FRR–jailbreak trade-off that l.362–366 argue against.
  • [10] Wang et al., Chinese Do-Not-Answer (2402.12193): a Chinese benchmark with false-positive (over-refusal) scenarios, relevant to the Mandarin baseline.
  • [11] Dong et al., Linguistic Discrimination in LLMs (IJCAI; 2404.18534): joint safety and response-quality evaluation in low-resource languages.
  • [12] Huang et al., Tower of Babel Revisited (2505.12287): Chinese ASR exceeds English on closed models. This conflicts with the near-equal 2.86% vs. 3.63% here.
  • [13] Choi, Lim & Kim, Culturally-Adapted Red-Teaming (2606.09178): direct translation underestimates risk in 44 of 48 cells, which matters for translated AdvBench.
  • [14] Shukla et al., Lost in Translation? Composite Harms (2602.07963): translated benchmarks are "necessary but not sufficient".
  • [15] Song et al., Multilingual Blending (2407.07342): mixed-language input raises bypass rates, relevant to Cantonese code-switching (l.102).
  • [16] Pattnayak & Chowdhuri, IndicJR (EACL 2026; 2602.16832): orthography and tokenization effects on ASR, relevant to the choice of Standard Written Chinese for the Cantonese instruction (l.156).

Additionally automated literature review also flagged Jain et al., PolygloToxicityPrompts (COLM 2024); Peppin et al., The Multilingual Divide (2505.21344); Tasawong et al., SEA-SafeguardBench (2512.05501); Sun et al., Safety Assessment of Chinese LLMs (2304.10436); Verma & Bharadwaj, The Hidden Space of Safety (2504.02708); Banerjee et al., Soteria (EMNLP 2025). The reference list has only 20 entries, which is short given that the novelty claims depend on what prior work has done. AdvBench (Zou et al.) is cited as an arXiv preprint only.

Questions for authors
  • How is a model's "English–Mandarin baseline" defined: the mean, the max, or each language separately?
  • Were English and Mandarin off-topic labels audited? Recompute Mandarin ASR with the same correction as Cantonese: does "every model higher in Cantonese" still hold?
  • Recompute ASR with empty and error outputs excluded rather than counted as refusals. How much do Claude and GPT-5.4 Tibetan and Qwen3.6 constrained Tibetan change?
  • Split Cantonese TLRR by outcome. Are the non-Cantonese AdvBench responses mostly refusals in Mandarin, Standard Written Chinese or English?
  • Which judge and adjudicator models were used? What are the judge–judge agreement rates per language, and how often was the adjudicator invoked?
  • How many human annotators labelled each language, and what is their inter-annotator agreement? Only one co-author is named for the Tibetan translation check (l.147).
    • Follow-up: who performed the human label audits (l.192–215, App. A.2)? Were they native speakers? The paper names co-authors only for the translation checks (l.142, l.147–149).
  • Which model and version was behind the deepseek-reasoner endpoint at your access date?
  • What are the decoding settings and samples per prompt? Can you give CIs on ASR and FRR and test the differences between correlation coefficients?
  • How severe or actionable is Tibetan "harmful compliance"? Would a StrongREJECT-style score change the picture?
  • How were the external AdvBench translations produced (machine translation + post-edit?), and are the Cantonese prompts in colloquial written Cantonese or Standard Written Chinese? Why is the Cantonese constraint instruction in Standard Written Chinese?
  • Why use 2023-era open models (Llama-2, Chinese-Alpaca) when the open-weight DeepSeek, GLM, Kimi, MiniMax and MiMo models could be run locally?
  • What is the "one response per model–language pair reassigned" in Tab. 2, and why?
  • Is there a per-harm-category ASR breakdown, for example to see whether the Tibetan gap is concentrated in specific categories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.