acceptodds
Under review as a conference paper at ICLR 2027

Beyond ASR: Evaluating Safety Alignment of Multimodal Large Language Models Across Modalities and Low-Resource Languages

Abstract

Safety alignment in multimodal large language models (MLLMs) may vary across languages, modalities, and underlying language capabilities. We evaluate multi- ple MLLMs on harmful and benign safety-sensitive prompts in English, Man- darin, Cantonese, and Tibetan under both direct-text and image-embedded instruc- tion settings. Beyond attack success rate (ASR), we jointly analyze over-refusal rate (ORR), target-language response rate, off-topic behavior, and visual instruc- tion recovery to distinguish genuine safety behavior from failures in low-resource language understanding or multimodal processing. Our results reveal substantial safety disparities in Cantonese and Tibetan, while showing that image embedding does not uniformly increase jailbreak risk. In particular, low ASR for Tibetan image inputs is often accompanied by high off-topic rates and poor recovery of the embedded instruction, indicating that capability failure can be mistaken for stronger safety. Cross-language image–text ablations further confirm that the lan- guage carried by the visual modality substantially affects model behavior. These findings highlight the importance of capability-aware evaluation for multilingual multimodal safety.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Large Language Models, Safety Alignment, Multilingual Safety, Low-Resource Languages, Safety Evaluation, Over-Refusal, Cross-Modal Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SarKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a multilingual, multimodal safety evaluation of MLLMs across English, Mandarin, Cantonese, and Tibetan under direct-text and image-embedded instructions. Beyond attack success rate (ASR) on AdvBench and over-refusal rate (ORR) on XSTest/OR-Bench Hard, the authors jointly measure target-language response rate (TLRR), off-topic rates, visual instruction recovery, and cross-language image–text ablations to separate genuine safety behavior from capability failures. Key findings: under text input, Cantonese and Tibetan show large safety gaps (mean ASR 21.43%/26.42% vs 1.30% for English); image embedding does not uniformly increase jailbreak risk — in fact Tibetan image ASR is low but accompanied by very high off-topic rates (up to ~74%) and near-zero instruction-recovery overlap, indicating that capability failure is being mistaken for strong safety; the language carried by the image dominates task failure; and ASR/ORR show no consistent trade-off.

Critical strengths
  • Important methodological contribution: the paper convincingly demonstrates that low ASR in low-resource multimodal settings can be an artifact of language/multimodal capability failure rather than successful safety alignment — a caveat the multilingual safety literature largely ignores, and one with direct implications for how safety evaluations should be reported.
  • Rich, multi-dimensional and well-controlled design: ASR/ORR are complemented by TLRR, off-topic analysis, forced-language conditions, instruction-recovery control experiments, and cross-language image–text ablations, which cleanly disentangle language effects from modality effects.
  • Covers genuinely understudied languages (Cantonese, Tibetan) with native-speaker review of translations, manual validation of model-based classifications, and careful ethics/reproducibility statements including controlled release of jailbreak data.
Critical weaknesses
  • The paper is purely diagnostic: it proposes no method, no mitigation, and no actionable guidance beyond "evaluate jointly"; the core findings (multilingual safety gaps, modality dependence) extend established results, with the novel part being the Tibetan/Cantonese multimodal setting and the capability-confound analysis.
  • Statistical and measurement reliability concerns: AdvBench has only 520 prompts, so ASR differences in low-rate cells rest on a handful of successes; no confidence intervals are reported; off-topic classification for Tibetan/Cantonese outputs depends on judge models whose own low-resource capability may be limited, and the reliability of this classification is not quantified in the main text.
  • Translation validity is a potential confound for the central claim: the Cantonese/Tibetan AdvBench translations come from an unnamed external team, and translation artifacts (non-idiomatic phrasing, changed implied harm) could themselves inflate the cross-lingual ASR gap; no back-translation or human validation of the harmful sets is reported.
  • The instruction-recovery overlap metric is a weak proxy for visual-language understanding (acknowledged), no mechanism-level analysis is offered (OCR, tokenizer, training-data coverage), and single-turn API evaluations with moving model versions limit reproducibility over time.
Questions for authors
  • How were the Cantonese/Tibetan harmful-prompt translations validated for quality and idiomaticity, and could translation artifacts explain part of the cross-lingual ASR differences rather than safety-alignment gaps?
  • Can you report confidence intervals for ASR/ORR (especially low-rate cells) and inter-annotator/judge agreement for the off-topic category in Tibetan, where the judge models' own capability is limited?
  • Do you have hypotheses for the mechanism behind Tibetan image failures (OCR, tokenizer, font rendering, training data)? Have you probed, e.g., different Tibetan fonts or Latin transliteration?
  • What concrete recommendations (e.g., capability-normalized ASR, guardrail adjustments, evaluation protocols) would you give practitioners and benchmark designers, given that the paper stops at diagnosis?
2 comments
AuthorsOpenReview ↗

Additional Validation, Robustness, and Practical Implications

We thank the reviewer for recognizing the methodological contribution and the value of our multidimensional evaluation. We address the concerns below and will incorporate the additional evidence and clarifications into the revised manuscript.

1. Translation quality and semantic equivalence.

The Cantonese and Tibetan AdvBench datasets were provided by an independent research team (Huang et al., Findings of EMNLP 2026), whose paper documents the translation procedures and native-speaker verification. The work was unpublished when our manuscript was prepared, and we will now add the formal citation and validation details. For our benign over-refusal datasets, the Cantonese translations were verified by two native Cantonese speakers, while the Tibetan translations were verified by one native Tibetan speaker due to personnel limitations. We acknowledge that native-speaker verification reduces, but does not completely eliminate, potential translation artifacts.

2. Statistical and classification reliability.

We will supplement ASR/ORR results with 95% confidence intervals, particularly for low-rate settings, and clarify the exploratory nature of model-level correlations.

Our evaluation uses Gemini-3.1-Flash-Lite-Preview and GPT-5.4-mini as independent judges, with Gemini-3.5-Flash adjudicating disagreements. For Tibetan harmful requests, initial judge agreement was 80.83% for text and 86.53% for images; for benign requests, the corresponding rates were 76.26% and 88.16%. Manual verification confirmed 48/50 (96%) Tibetan harmful off-topic predictions and 45/50 (90%) benign off-topic predictions. We will report these statistics and acknowledge that judge agreement does not independently establish classification accuracy.

3. Mechanisms underlying Tibetan image failures.

Our Tibetan images use Noto Serif Tibetan, a 36-pixel font size, and syllable-aware line wrapping. Manual inspection confirmed faithful rendering in all 50 sampled images.

To better characterize visual instruction recovery, we additionally computed Unicode-normalized character error rate (CER) alongside strict exact matching. Tibetan exhibits a median model CER of 44.20%, compared with 7.16% for Cantonese, 1.12% for English, and 0.60% for Mandarin. However, performance varies substantially across models (e.g., Grok-4.5: 3.89%; GLM-5V-Turbo: 76.08%). These results reveal heterogeneous transcription difficulties rather than a universal Tibetan OCR failure. We have not yet isolated the effects of fonts, tokenization, or training-data coverage, and will explicitly distinguish these possible mechanisms from our observed capability-related failures.

4. Additional insights and practical recommendations.

Our additional response-language-conditioned analysis reveals that language fallback does not universally improve safety. For example, GPT-5.6 exhibits 20.22% Cantonese TL-ASR versus 43.14% non-target-language ASR under text input, while Qwen3.7-Plus exhibits 22.44% TL-ORR versus 70.56% non-target-language ORR under Cantonese image input.

Furthermore, GLM-5V-Turbo achieves 99.81% Tibetan target-language response rate but 0% TL-ASR and 100% off-topic responses on harmful image inputs, demonstrating that target-language adherence alone does not guarantee instruction comprehension.

For practitioners, we recommend reporting ASR and ORR alongside target-language adherence, off-topic rates, and explicit failure categories, with language-conditioned and modality-specific breakdowns. Visual instruction recovery should be evaluated separately before interpreting low image-based ASR as strong safety alignment.

We will incorporate these additional analyses, uncertainty estimates, evaluation recommendations, and limitations into the revised manuscript.

I appreciate the authors' efforts. The author’s response has indeed given me a better understanding of the paper; however, these revisions are not sufficient to warrant a higher score, so I am maintaining my original rating.

Reviewer_q84qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper evaluates the safety alignment of nine MLLMs across English, Mandarin, Cantonese, and Tibetan, using harmful and benign prompts under text and image-embedded settings. Beyond ASR and ORR, it examines target-language response rates, off-topic behavior, and visual instruction recovery. The results show higher ASR in Cantonese and Tibetan and reveal that low ASR for Tibetan image inputs often coincides with poor instruction recovery and high off-topic rates. The paper argues that capability failures can be mistaken for stronger safety alignment.

Critical strengths
  1. The paper demonstrates that low ASR may reflect instruction-understanding failures rather than genuine safety alignment, particularly in low-resource languages.
  2. Evaluations across nine models, four languages, and multiple input settings provide useful evidence of language- and modality-dependent safety behavior.
Critical weaknesses
  1. The visual instruction-recovery metric is not adequately specified or validated. Textual overlap alone cannot clearly distinguish OCR errors from semantic misunderstanding. Judge-model details and classification reliability also need clarification. Moreover, Appendix A’s discussion of Tibetan off-topic classification appears inconsistent with the confirmation rates in Table 1.
  2. Limited cross-language validation and statistical analysis. Translation quality and semantic equivalence are insufficiently established, while uncertainty estimates and statistical comparisons are limited. These issues weaken confidence in the robustness and generality of the findings.
  3. The central conclusion—that ASR alone is insufficient for evaluating safety without considering model capabilities—is intuitive and somewhat expected. Although the experiments provide illustrative evidence, the evaluation in each setting relies on only several hundred prompts. The paper could be more convincing by systematically investigating additional factors that may affect safety evaluation and disentangling their respective contributions, thereby providing deeper insights and a more substantial contribution beyond confirming this intuitive observation.
Questions for authors
  1. How exactly is visual instruction recovery measured, and can it be validated against an independent OCR or instruction-reading baseline? Could the authors clarify the apparent inconsistency between Appendix A and Table 1?
  2. How were Cantonese and Tibetan translations validated for semantic equivalence? Will the data be released publicly?
2 comments
AuthorsOpenReview ↗

Additional Validation and Clarification of Contributions

We thank the reviewer for the constructive feedback and for recognizing the value of our multilingual and multimodal evaluation. We address the concerns below and will incorporate the additional analyses and clarifications into the revised manuscript.

1. Visual instruction recovery and evaluation reliability.

We clarify that the originally reported "overlap" measures strict full-string exact matching. To address its limitations, we additionally computed character error rate (CER) using Unicode NFC-normalized Levenshtein distance. The median model CER for Tibetan is 44.20%, compared with 7.16% for Cantonese, 1.12% for English, and 0.60% for Mandarin. However, Tibetan recovery varies substantially across models (e.g., Grok-4.5: 3.89%; GLM-5V-Turbo: 76.08%). We will report these results and clarify that transcription fidelity alone cannot distinguish OCR errors from semantic misunderstanding. An independent Tibetan-capable OCR baseline remains an important additional validation.

For response classification, we employed Gemini-3.1-Flash-Lite-Preview and GPT-5.4-mini as independent judges, with Gemini-3.5-Flash adjudicating disagreements. Initial judge agreement on Tibetan harmful requests was 80.83% for text and 86.53% for images. Our manual audit further confirmed 48/50 (96%) Tibetan harmful off-topic predictions and 45/50 (90%) benign off-topic predictions.

We also acknowledge the inconsistency between Appendix A and Table 1. The numerical results in Table 1 are correct; the accompanying description overstated the unreliability of Tibetan off-topic classification. We will correct this wording and explicitly report the validation procedures.

2. Translation quality, statistical robustness, and data availability.

The Cantonese and Tibetan AdvBench datasets were provided by an independent research team (Huang et al., Findings of EMNLP 2026), whose paper documents their construction and native-speaker verification. The source paper was unpublished when our manuscript was prepared. We will add the formal citation and validation details.

Our benign over-refusal translations were reviewed by two native Cantonese speakers and one native Tibetan speaker, respectively. We acknowledge the limitation of single-speaker Tibetan verification.

We will also add 95% confidence intervals for ASR/ORR and clarify the exploratory nature of correlations based on 8–9 models.

The original translated AdvBench datasets are now publicly available through Huang et al. We will link to their release. Our benign over-refusal datasets will be publicly released upon acceptance, as stated in our manuscript.

3. Additional insights beyond ASR-based evaluation.

We further analyzed ASR and ORR conditioned on target-language and non-target-language responses. Language fallback does not universally reduce harmful compliance: GPT-5.6 exhibits 20.22% Cantonese TL-ASR versus 43.14% non-target-language ASR under text input. Conversely, Qwen3.7-Plus shows 22.44% TL-ORR versus 70.56% non-target-language ORR under Cantonese image input.

Moreover, GLM-5V-Turbo achieves 99.81% Tibetan target-language response rate but 0% TL-ASR and 100% off-topic responses on harmful image inputs.

These findings extend prior text-only studies by showing that response-language behavior, over-refusal, and visual-language capability jointly affect multimodal safety evaluation. We will incorporate these additional analyses and sharpen the distinction between our empirical contributions and prior work.

Reviewer_q84qOpenReview ↗

Response

Thank you for your response. In my view, the academic contributions of this work remain limited, so I keep my original score.

Reviewer_hP6VOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper evaluates multimodal LLMs on translated AdvBench, XSTest, and OR-Bench-Hard prompts in English, Mandarin, Cantonese, and Tibetan. They test under two settings: image-as-text and image-embedded. They advocate for reporting over-refusal rates alongside ASR, along with multi-lingual/multimodal specific metrics like target-language response rate, off-topic rate, and an image-instruction recovery check. The main claim is that low Tibetan image ASR largely reflects comprehension failure rather than safety.

Critical strengths
  1. The paper focuses on multilingual safety that is generally under explored, specifically Cantonese and Tibetan
  2. The cross-language image/text ablation shows a clean, intuitive result: Tibetan in the image produces 65% off-topic responses, compared with 16% when Tibetan is in the text
Critical weaknesses
  1. The central message, that low-resource safety can be incomprehension, is already established. So is the idea to look beyond just "Attack Success Rates" and at over-refusal rates. Several benchmarks have been introduced precisely to represent this tradeoff. Yong et al. (2023) and Deng et al. (2024) both discuss unclear or invalid outputs in low-resource jailbreaks, and StrongREJECT targets exactly this empty jailbreak problem. The paper offers no new metric, method, or mitigation beyond reporting an off-topic rate.
  • Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. Low-Resource Languages Jailbreak GPT-4. NeurIPS Workshop on Socially Responsible Language Modelling Research, 2023. arXiv:2310.02446
  • Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual Jailbreak Challenges in Large Language Models. ICLR 2024.
  • Alexandra Souly et al. A StrongREJECT for Empty Jailbreaks. NeurIPS 2024, Datasets and Benchmarks Track
  1. Annotation and data quality is underspecified and under-validated. The Cantonese and Tibetan AdvBench translations come from an unnamed external team with no reported validation. Only the over-refusal set was reviewed by coauthors, and the Tibetan review was done by a single person. The judge models are not named and there is no assessment whether the judge itself is good at the target language. Pearson correlation in results is computed over 8–9 points and swings from 0.88 to 0.60 depending on the refusal definition. They support no conclusion.
Questions for authors
  1. Which models served as judges and adjudicator, and how was their Tibetan competence verified?
  2. How is “overlap” computed for Tibetan?
2 comments
AuthorsOpenReview ↗

Clarifications and Additional Experimental Evidence

We thank the reviewer for the constructive feedback. We acknowledge that the original manuscript insufficiently documented several methodological details and will address these issues in the revision.

W1. Novelty and contributions. We agree that the limitations of ASR and low-resource language comprehension failures have been previously studied. Our contribution lies in systematically examining how input modality, response language, and visual instruction recovery jointly affect multimodal safety evaluation, rather than proposing a new standalone metric.

Beyond prior text-only studies, our cross-language ablation reveals substantially higher off-topic rates when Tibetan appears in images (65.19%) than in accompanying text (16.47%). We further conducted response-language-conditioned analyses of both ASR and ORR. For example, GPT-5.6 exhibits lower Cantonese TL-ASR (20.22%) than non-target-language ASR (43.14%), showing that language fallback does not universally reduce harmful compliance. Meanwhile, Qwen3.7-Plus exhibits substantially higher non-target-language ORR (70.56%) than TL-ORR (22.44%) on Cantonese image inputs. These findings reveal modality-dependent safety behaviors and potential over-refusal costs overlooked by ASR-only evaluations. We will incorporate these additional analyses into the revised manuscript.

W2. Dataset provenance and validation. The Cantonese and Tibetan AdvBench datasets were provided by an independent research team (Huang et al., Findings of EMNLP 2026), whose paper documents their construction and native-speaker verification. As this work was unpublished when our manuscript was prepared, we originally described the datasets as externally provided. We will add the formal citation and validation details.

For our benign over-refusal datasets, the Cantonese translations were verified by two native Cantonese speakers, while the Tibetan translations were verified by one native Tibetan speaker due to limited personnel. We acknowledge this limitation and will explicitly document it.

Q1. Judge models and reliability. We employed Gemini-3.1-Flash-Lite-Preview and GPT-5.4-mini as independent judges, with Gemini-3.5-Flash adjudicating disagreements. All individual judgments and arbitration records have been retained. Initial judge agreement on Tibetan harmful requests was 80.83% for text and 86.53% for images. Our manual audit further confirmed 50/50 sampled harmful-compliance and refusal predictions, as well as 48/50 harmful off-topic and 45/50 benign off-topic predictions. Although agreement alone does not establish judge accuracy, these manual checks provide complementary evidence of classification reliability. We will report the judging procedure, agreement statistics, and validation limitations in the revision.

Q2. Visual instruction overlap. The reported overlap is a strict full-string exact-match rate. To address its limitations, we additionally computed character error rate (CER) using Unicode NFC-normalized Levenshtein distance. For matched-language image recovery, Tibetan has a mean exact-match rate of 4.93% and a median model CER of 44.20%. CER also reveals substantial model-level variation, demonstrating that low exact-match rates do not uniformly indicate poor character recognition. We will clarify the metric definition, add CER comparisons across languages, and distinguish transcription failure from semantic misunderstanding.

Statistical robustness. We agree that Pearson correlations based on 8–9 models are sensitive to metric definitions. We will treat these correlations as exploratory, report their limitations, and avoid drawing universal trade-off conclusions from them.

Thank you for your response. I keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.