- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a multilingual, multimodal safety evaluation of MLLMs across English, Mandarin, Cantonese, and Tibetan under direct-text and image-embedded instructions. Beyond attack success rate (ASR) on AdvBench and over-refusal rate (ORR) on XSTest/OR-Bench Hard, the authors jointly measure target-language response rate (TLRR), off-topic rates, visual instruction recovery, and cross-language image–text ablations to separate genuine safety behavior from capability failures. Key findings: under text input, Cantonese and Tibetan show large safety gaps (mean ASR 21.43%/26.42% vs 1.30% for English); image embedding does not uniformly increase jailbreak risk — in fact Tibetan image ASR is low but accompanied by very high off-topic rates (up to ~74%) and near-zero instruction-recovery overlap, indicating that capability failure is being mistaken for strong safety; the language carried by the image dominates task failure; and ASR/ORR show no consistent trade-off.
Critical strengths
- Important methodological contribution: the paper convincingly demonstrates that low ASR in low-resource multimodal settings can be an artifact of language/multimodal capability failure rather than successful safety alignment — a caveat the multilingual safety literature largely ignores, and one with direct implications for how safety evaluations should be reported.
- Rich, multi-dimensional and well-controlled design: ASR/ORR are complemented by TLRR, off-topic analysis, forced-language conditions, instruction-recovery control experiments, and cross-language image–text ablations, which cleanly disentangle language effects from modality effects.
- Covers genuinely understudied languages (Cantonese, Tibetan) with native-speaker review of translations, manual validation of model-based classifications, and careful ethics/reproducibility statements including controlled release of jailbreak data.
Critical weaknesses
- The paper is purely diagnostic: it proposes no method, no mitigation, and no actionable guidance beyond "evaluate jointly"; the core findings (multilingual safety gaps, modality dependence) extend established results, with the novel part being the Tibetan/Cantonese multimodal setting and the capability-confound analysis.
- Statistical and measurement reliability concerns: AdvBench has only 520 prompts, so ASR differences in low-rate cells rest on a handful of successes; no confidence intervals are reported; off-topic classification for Tibetan/Cantonese outputs depends on judge models whose own low-resource capability may be limited, and the reliability of this classification is not quantified in the main text.
- Translation validity is a potential confound for the central claim: the Cantonese/Tibetan AdvBench translations come from an unnamed external team, and translation artifacts (non-idiomatic phrasing, changed implied harm) could themselves inflate the cross-lingual ASR gap; no back-translation or human validation of the harmful sets is reported.
- The instruction-recovery overlap metric is a weak proxy for visual-language understanding (acknowledged), no mechanism-level analysis is offered (OCR, tokenizer, training-data coverage), and single-turn API evaluations with moving model versions limit reproducibility over time.
Questions for authors
- How were the Cantonese/Tibetan harmful-prompt translations validated for quality and idiomaticity, and could translation artifacts explain part of the cross-lingual ASR differences rather than safety-alignment gaps?
- Can you report confidence intervals for ASR/ORR (especially low-rate cells) and inter-annotator/judge agreement for the off-topic category in Tibetan, where the judge models' own capability is limited?
- Do you have hypotheses for the mechanism behind Tibetan image failures (OCR, tokenizer, font rendering, training data)? Have you probed, e.g., different Tibetan fonts or Latin transliteration?
- What concrete recommendations (e.g., capability-normalized ASR, guardrail adjustments, evaluation protocols) would you give practitioners and benchmark designers, given that the paper stops at diagnosis?
2 comments
I appreciate the authors' efforts. The author’s response has indeed given me a better understanding of the paper; however, these revisions are not sufficient to warrant a higher score, so I am maintaining my original rating.
Additional Validation, Robustness, and Practical Implications
We thank the reviewer for recognizing the methodological contribution and the value of our multidimensional evaluation. We address the concerns below and will incorporate the additional evidence and clarifications into the revised manuscript.
1. Translation quality and semantic equivalence.
The Cantonese and Tibetan AdvBench datasets were provided by an independent research team (Huang et al., Findings of EMNLP 2026), whose paper documents the translation procedures and native-speaker verification. The work was unpublished when our manuscript was prepared, and we will now add the formal citation and validation details. For our benign over-refusal datasets, the Cantonese translations were verified by two native Cantonese speakers, while the Tibetan translations were verified by one native Tibetan speaker due to personnel limitations. We acknowledge that native-speaker verification reduces, but does not completely eliminate, potential translation artifacts.
2. Statistical and classification reliability.
We will supplement ASR/ORR results with 95% confidence intervals, particularly for low-rate settings, and clarify the exploratory nature of model-level correlations.
Our evaluation uses Gemini-3.1-Flash-Lite-Preview and GPT-5.4-mini as independent judges, with Gemini-3.5-Flash adjudicating disagreements. For Tibetan harmful requests, initial judge agreement was 80.83% for text and 86.53% for images; for benign requests, the corresponding rates were 76.26% and 88.16%. Manual verification confirmed 48/50 (96%) Tibetan harmful off-topic predictions and 45/50 (90%) benign off-topic predictions. We will report these statistics and acknowledge that judge agreement does not independently establish classification accuracy.
3. Mechanisms underlying Tibetan image failures.
Our Tibetan images use Noto Serif Tibetan, a 36-pixel font size, and syllable-aware line wrapping. Manual inspection confirmed faithful rendering in all 50 sampled images.
To better characterize visual instruction recovery, we additionally computed Unicode-normalized character error rate (CER) alongside strict exact matching. Tibetan exhibits a median model CER of 44.20%, compared with 7.16% for Cantonese, 1.12% for English, and 0.60% for Mandarin. However, performance varies substantially across models (e.g., Grok-4.5: 3.89%; GLM-5V-Turbo: 76.08%). These results reveal heterogeneous transcription difficulties rather than a universal Tibetan OCR failure. We have not yet isolated the effects of fonts, tokenization, or training-data coverage, and will explicitly distinguish these possible mechanisms from our observed capability-related failures.
4. Additional insights and practical recommendations.
Our additional response-language-conditioned analysis reveals that language fallback does not universally improve safety. For example, GPT-5.6 exhibits 20.22% Cantonese TL-ASR versus 43.14% non-target-language ASR under text input, while Qwen3.7-Plus exhibits 22.44% TL-ORR versus 70.56% non-target-language ORR under Cantonese image input.
Furthermore, GLM-5V-Turbo achieves 99.81% Tibetan target-language response rate but 0% TL-ASR and 100% off-topic responses on harmful image inputs, demonstrating that target-language adherence alone does not guarantee instruction comprehension.
For practitioners, we recommend reporting ASR and ORR alongside target-language adherence, off-topic rates, and explicit failure categories, with language-conditioned and modality-specific breakdowns. Visual instruction recovery should be evaluated separately before interpreting low image-based ASR as strong safety alignment.
We will incorporate these additional analyses, uncertainty estimates, evaluation recommendations, and limitations into the revised manuscript.