acceptodds
Withdrawn from ICLR 2027

Beyond Translation: Evaluating Language-Conditioned Safety Misalignment in LLMs Across Cantonese and Tibetan

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multilingual safety evaluation often relies on translated benchmarks, making it unclear whether observed outcomes reflect safety alignment or language-specific behavior. We evaluate ten large language models on Cantonese and nine on Tibetan across jailbreak and over-refusal settings, comparing translated benchmarks with language-specific datasets under both natural and forced target-language response conditions. Beyond attack success rate (ASR), we measure target-language response rate (TLRR), target-language-conditioned ASR, explicit textual overrefusal, and off-topic behavior. Our results show that aggregate ASR alone can be misleading, as low attack success may arise either from stronger safety alignment or from failures to respond in the requested language. We further find that translated and language-specific datasets can yield substantially different conclusions: ASR is consistently higher on the language-specific Cantonese benchmark, whereas the difference between translated and language-specific Tibetan benchmarks is more model-dependent. In addition, the relationship between jailbreak robustness and over-refusal varies across languages and datasets rather than forming a universal trade-off. These findings suggest that multilingual safety assessment should jointly consider safety alignment and target-language adherence, and that language-specific evaluation provides complementary insights beyond translated benchmarks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multilingual LLM Safety, Language-Conditioned Alignment, Jailbreak Robustness, Over-Safety, Target-Language Response, Cantonese, Tibetan, Safety Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iEYvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Multilingual safety evaluation can be confounded by language adherence. This work proposes human-validated, language-specific jailbreak and over-refusal benchmarks for Cantonese and Tibetan, and runs extensive experiments on ten LLMs, comparing them with translated benchmarks while taking the bias from language adherence into account.

Critical strengths

S1. Valuable benchmarks for low-resource languages. Building safety datasets for Cantonese and Tibetan with native-speaker review makes the data rare and valuable for future research.

S2. Useful findings. Simply asking for an answer in the target language can sharply raise ASR, even when the model already answers in that language. The evaluation also reveals access problems in Tibetan: Kimi-K2.6 rejects Tibetan input entirely, and DeepSeek-V4-Pro gives no answer to 73% of translated Tibetan benign prompts, mostly as blank outputs.

Critical weaknesses

W1. The forced-setting results do not yet support the TL-ASR claim. Refusals are far more often in a non-target language than harmful answers are (Grok-4.3, translated Tibetan: 13% vs 91% in Tibetan), and when forcing sharply raises TLRR, ASR barely changes (Grok: TLRR 43.9 → 81.7%, ASR 29.8 → 31.0%), so low ASR is not shown to reflect poor language adherence. In addition, blank outputs count as refusals in ASR but are dropped from the TL-ASR denominator, which inflates TL-ASR.

W2. Most comparisons change several factors at once. The translated prompts are plain AdvBench requests, while the language-specific sets add jailbreak wrappers and different harm taxonomies; the forcing instruction also changes the prompt itself, not just the response language (GPT-5.5: TLRR 99.8 → 100%, ASR 10.6 → 38.4%). Matched controls, such as the same request with and without a wrapper and a length-matched neutral instruction, are needed before these effects can be attributed to language.

W3. No source-language baseline. Without results on the original English or Chinese prompts (e.g., English AdvBench, or the Chinese renderings the Tibetan prompts already have), the paper cannot show whether translated benchmarks misrepresent minority-language safety, which is its stated aim.

Questions for authors
  1. For each model, what share of harmful responses vs refusals is in the target language? Which non-target languages appear? How do you see the Grok-4.3 and Mimo forced-setting results fitting the language-switching interpretation?

  2. Could you break down ASR on the language-specific sets by jailbreak strategy and harm category? Could you add a comparison of the same harmful request with and without a wrapper?

  3. Cantonese is a low-resource language but not a lesser-spoken one. Did you try to find more native-speaker reviewers to make the checks statistically sound?

Reviewer_zkhXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates 10 frontier API models on Cantonese and Tibetan. It covers jailbreak and over-refusal. Each jailbreak set is run "natural" and "forced", where forced is natural + appended an instruction to answer in the target language. The central finding is that aggregate ASR mixes up safety with failing to answer in the requested language. Interstingly, translated and language-specific benchmarks lead to different conclusions: ASR is always higher on the Cantonese language-specific set, and mostly lower on the Tibetan one. The author report an inconsistent trade-off between robustness and over-refusal.

Critical strengths
  • relevant topic. evaluating safety in a multilingual setting is non-trivial and goes well beyond mere translation. specifically, when scripts change.
  • The language pair is well chosen. Cantonese shares a script with Mandarin and Tibetan has its own, which separates surface transfer from language-specific effects
  • The authors are very meticulous generally, e.g. with response in the wrong language and safety refusal vs. generally missing capability.
Critical weaknesses
  • the findings are very interesting, yet their scope is very limited. It's super unclear how well those findings generalize across other languages and scripts.
  • the model evaluated are very chinese-english heavy, specifically in the multilingual/cross-lingual interest, it is super interesting to include models like the latest mistral or kolibri version.
  • its a bit unclear to me what the learnings are now, what do we learn from those evals, how should we change in the future?
  • the eval is pretty much focused on low-resource languages, I think this is an issue because it limits the transfer to high resource cases.
Questions for authors

can you please provide more information on the human verification?

Reviewer_1YBKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that multilingual LLM safety cannot be reliably evaluated using translated benchmarks and aggregate attack success rate alone. Evaluating Cantonese and Tibetan, this paper shows that apparent jailbreak robustness can be confounded by whether models actually respond in the requested language. They also find that language-specific jailbreak and over-refusal benchmarks can produce substantially different conclusions from translated benchmarks. Overall, the paper recommends separately evaluating safety alignment, target-language adherence, over-refusal, and off-topic behavior.

Critical strengths
  • The multilingual safety topic is critical.
Critical weaknesses
  • The figures look noticeably AI-generated, with generic visuals and excessive decorative elements / textual explanations. I recommend redrawing them in a cleaner academic style.
  • The human validation details are extremely unclear. Since this work involves both collecting and validating data from human native speakers, including those details is highly critical for understanding the validity of the contribution. Also, the scale of human validation is also important. Minimal number of involved human annotators can introduce data artifacts and biases. So the annotation should go beyond the narrow annotation simply among handful of authors.
  • There's lack of citations in the intro for backing up claims.
  • The method descriptions are all very vague. For instance, it's unclear what's the source dataset, how translations are done, how verifications are done.
  • There's lack of concrete sense of what's qualitative difference between translated and native-speaker author's safety prompts.
  • There's significant reliance on LLM judges. For instance, using them to classify model responses. Is there any validation of these LLM judges?
Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.