- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Multilingual safety evaluation can be confounded by language adherence. This work proposes human-validated, language-specific jailbreak and over-refusal benchmarks for Cantonese and Tibetan, and runs extensive experiments on ten LLMs, comparing them with translated benchmarks while taking the bias from language adherence into account.
Critical strengths
S1. Valuable benchmarks for low-resource languages. Building safety datasets for Cantonese and Tibetan with native-speaker review makes the data rare and valuable for future research.
S2. Useful findings. Simply asking for an answer in the target language can sharply raise ASR, even when the model already answers in that language. The evaluation also reveals access problems in Tibetan: Kimi-K2.6 rejects Tibetan input entirely, and DeepSeek-V4-Pro gives no answer to 73% of translated Tibetan benign prompts, mostly as blank outputs.
Critical weaknesses
W1. The forced-setting results do not yet support the TL-ASR claim. Refusals are far more often in a non-target language than harmful answers are (Grok-4.3, translated Tibetan: 13% vs 91% in Tibetan), and when forcing sharply raises TLRR, ASR barely changes (Grok: TLRR 43.9 → 81.7%, ASR 29.8 → 31.0%), so low ASR is not shown to reflect poor language adherence. In addition, blank outputs count as refusals in ASR but are dropped from the TL-ASR denominator, which inflates TL-ASR.
W2. Most comparisons change several factors at once. The translated prompts are plain AdvBench requests, while the language-specific sets add jailbreak wrappers and different harm taxonomies; the forcing instruction also changes the prompt itself, not just the response language (GPT-5.5: TLRR 99.8 → 100%, ASR 10.6 → 38.4%). Matched controls, such as the same request with and without a wrapper and a length-matched neutral instruction, are needed before these effects can be attributed to language.
W3. No source-language baseline. Without results on the original English or Chinese prompts (e.g., English AdvBench, or the Chinese renderings the Tibetan prompts already have), the paper cannot show whether translated benchmarks misrepresent minority-language safety, which is its stated aim.
Questions for authors
-
For each model, what share of harmful responses vs refusals is in the target language? Which non-target languages appear? How do you see the Grok-4.3 and Mimo forced-setting results fitting the language-switching interpretation?
-
Could you break down ASR on the language-specific sets by jailbreak strategy and harm category? Could you add a comparison of the same harmful request with and without a wrapper?
-
Cantonese is a low-resource language but not a lesser-spoken one. Did you try to find more native-speaker reviewers to make the checks statistically sound?