- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates multilingual safety evaluation in multimodal large language models (MLLMs), focusing on whether low attack success or over-refusal rates may reflect failures in visual-text recognition, language following, or task understanding rather than stronger safety alignment. The authors evaluate nine MLLMs across English, Mandarin, Cantonese, and Tibetan using image-embedded jailbreak, over-safety, and neutral instructions. The evaluation jointly considers attack success rate (ASR), over-refusal rate (ORR), off-topic rate (OFF), and target-language reply rate (TLRR), supplemented by target-language enforcement, cross-lingual text–image mismatch, and neutral-image controls. The results reveal substantial language- and model-dependent differences. In particular, several models exhibit low ASR in Tibetan while producing predominantly off-topic responses and performing poorly on image-text reproduction tasks. The paper argues that multilingual multimodal safety should therefore be evaluated jointly with language compliance, task relevance, and visual-language understanding.
Critical strengths
The main strength is the empirical documentation of a potentially misleading pattern in multilingual multimodal safety evaluation: low attack success can coexist with poor task relevance and weak visual-text recognition. The evaluation across nine MLLMs, together with neutral-image and OCR controls, provides concrete examples of this issue, particularly for Tibetan.
However, I view this primarily as a useful diagnostic observation rather than a substantial conceptual or methodological contribution.
Critical weaknesses
The paper investigates a relevant evaluation concern, but its scientific contribution is substantially limited. The central observation—that low attack success may reflect failures in visual-text recognition rather than effective safety alignment—is largely intuitive, and the paper provides little additional insight beyond empirically demonstrating this possibility.
The analysis remains predominantly descriptive, without identifying a sufficiently novel evaluation principle, a deeper explanation of the observed behavior, or a broadly generalizable finding. Furthermore, despite evaluating multiple models, the experimental scope is narrow, and the study lacks direct controls necessary to substantiate its central disentanglement claim.
Taken together, these limitations concern both the significance of the contribution and the strength of the supporting evidence. I do not find the current submission to make a sufficiently substantial research contribution for acceptance at ICLR.
Questions for authors
Q1. Novelty and insights. Beyond the intuitive observation that visual recognition failures can reduce jailbreak success, what non-obvious or generalizable insights does this work provide?
Q2. Disentanglement validity. How do the experiments distinguish visual recognition failures from genuine safety refusals? Could the authors provide text-only versus image-embedded comparisons or evaluate safety behavior conditional on successful visual recognition?
Q3. Generalizability. Given the limited language coverage and reliance on image-rendered text, particularly the Tibetan results, how well do the findings generalize to other low-resource languages and realistic multimodal scenarios?
2 comments
Additional Cross-Linguistic Evidence (Q3)
We thank the reviewer for raising the generalizability concern.
We have additionally evaluated Russian text-only instructions on the corresponding AdvBench jailbreak (520 samples) and benign over-refusal (500 samples) datasets. Seven models were successfully evaluated; Claude was unavailable due to access restrictions, while MiMo-V2.5 was unavailable through its official service.
Using six models shared across all five languages (English, Mandarin, Cantonese, Tibetan, and Russian), the mean Russian ASR and ORR are 8.43% and 12.73%, compared with 27.53% and 15.20% for Tibetan. This provides additional evidence of language-dependent safety behavior.
Furthermore, response-language stratification reveals important differences. For example, Gemini-3.1-Pro's Russian ORR decreases from 9.40% overall to 2.37% among Russian-language responses, while all 36 non-Russian responses are refusals. Thus, aggregate safety metrics can obscure substantial language-conditioned differences.
We acknowledge that Russian is not a low-resource language and that these additional experiments are text-only. They therefore strengthen cross-linguistic evidence but do not establish generalization to other low-resource visual-language settings.
Regarding practical relevance, image-embedded instructions represent interactions involving screenshots, scanned documents, and visually presented text. We will clarify that our conclusions apply primarily to such image-text interactions rather than all multimodal scenarios.
Novel Insights and Disentanglement Evidence (Q1–Q2)
We thank the reviewer for the constructive feedback. We have conducted additional analyses using our existing experimental results.
Q1. Novel Insights Beyond Visual Recognition Failures
Our findings go beyond the observation that poor visual recognition can reduce jailbreak success. We identify a strong association between image-text reproduction and task relevance across models.
For Tibetan, exact-match reproduction rates for benign image text are strongly negatively correlated with OFF in both jailbreak (Pearson r = −0.854) and over-refusal (r = −0.839) settings. Similar associations emerge using neutral-text reproduction (r = −0.727 and −0.723, respectively; n = 8).
Importantly, these patterns differ across languages and models. This suggests that safety metrics may be systematically confounded by image-text processing capabilities, rather than reflecting safety alignment alone.
Q2. Text-Only vs. Image-Embedded Comparisons
We additionally analyzed existing text-only and image-embedded experiments using corresponding instructions and models.
For example, under Tibetan instructions:
These results reveal substantial model-dependent differences rather than a universal effect of image-embedded instructions.
We emphasize that the two settings use different task-appropriate prompts: text-only inputs present instructions directly, whereas image inputs explicitly ask models to process the embedded instructions. Thus, these comparisons provide complementary observational evidence, not a strictly controlled causal disentanglement.
Together with our benign and neutral-image controls, these findings strengthen the evidence that low ASR should not automatically be interpreted as successful safety refusal.