- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies implicit multimodal harm, where an image and text can be benign individually but harmful when combined. The authors introduce MuPHI, a benchmark designed to evaluate how well VLMs understand such harmful content. Their analysis shows that VLMs often struggle to connect information from images and text, even when they understand each separately. To address this, the authors propose CHIL, which trains VLMs to reason about harmful content rather than simply predict labels.
Critical strengths
-
Improving the cross-modal reasoning ability of VLMs is an important problem for multimodal safety and worth further investigation.
-
The analysis using counterfactual examples is useful. It helps identify whether VLMs actually understand how images and text interact to produce harmful meaning.
Critical weaknesses
-
My main concern is the limited novelty of MuPHI. The problem of benign images and text becoming harmful when combined has already been studied in VLM safety benchmarks such as SIUO [1] and ELITE [2]. MuPHI seems to mainly apply this idea to images with embedded text, but similar settings have already been explored in Hateful Memes [3], including counterfactual examples. Therefore, I find it difficult to see what is fundamentally new in MuPHI compared to existing benchmarks.
-
Another concern is the limited comparison with existing reasoning-based methods. Although the paper compares CHIL with SFT and outcome-only GRPO, it does not directly compare against recent methods [4]. Without these comparisons, it is difficult to assess whether CHIL provides meaningful improvements over existing approaches.
[1] Safe Inputs but Unsafe Output: Benchmarking Cross-Modal Safety Alignment in Vision-Language Models, NAACL 2025.
[2] Enhanced Language-Image Toxicity Evaluation for Safety, ICML 2025.
[3] The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes, NeurIPS 2020.
[4] ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection, ICLR 2026.
Questions for authors
-
Could the authors compare CHIL with recent reasoning-based methods? This would help clarify the contribution of CHIL over existing approaches.
-
Regarding Weakness 1, could the authors clarify what new challenges MuPHI introduces compared to existing benchmarks?
4 comments
Concern: CHIL lacks comparison to recent reasoning-based methods such as ExPO-HM (W2, Q1)
Table C:
| Method | MuPHI (in-domain) | FHM | Harm-C | Harm-P | OOD mean |
|---|---|---|---|---|---|
| ProCap | 89.6 | 49.4 | 41.0 | 34.9 | 41.8 |
| RGCL | 90.2 | 56.5 | 20.1 | 36.6 | 37.7 |
| SFT (structured prompt) | 91.6 | 58.7 | 66.0 | 54.1 | 59.6 |
| ExPO-HM | 79.8 | 61.7 | 57.1 | 59.1 | 59.3 |
| CHIL | 96.9 | 64.2 | 64.9 | 59.0 | 62.7 |
Comparison with ExPO-HM. We replicated ExPO-HM on MuPHI with the authors' released code and hyperparameters: policy-manual SFT, then curriculum GRPO with the CDE reward. We evaluated it with its own inference protocol on the same test items as all our tables. CHIL is better both in-domain (96.9 vs. 79.8 macro-F1) and on the unseen datasets (62.7 vs. 59.3 OOD mean).
Thank you for the detailed response. I appreciate the author's efforts to address my concerns.
While some of my concerns have been addressed, I still have concerns about the overall novelty of the paper. I understand the motivation behind this work and appreciate the effort to better understand implicit multimodal harm. However, I feel that the overall approach largely builds on existing research, and I am still not fully convinced that the paper offers sufficiently new insights or vulnerability.
Nevertheless, the additional clarifications and experimental results have addressed some of my concerns, so I am willing to raise my rating from 1 to 2.
We thank the reviewer for acknowledging our clarifications and for reconsidering their score. We have also included the clarifications in the updated paper (marked in blue in both main and supplementary) as suggested.
For additional clarity, we would like to further summarize the novelty and insights that MuPHI brings forth:
- MuPHI is a controlled testbed (free from niche external references, unlike other harm datasets) built to investigate whether models identify the cross-modal mechanism that results in the harmful interpretation, and not just whether they can classify into a binary label.
- Together with benign counterfactuals and human-annotated rationales in MuPHI, we provide insights that accuracy hides. VLMs ground and interpret each modality correctly but fail to bind them which yields the necessary harm interpretation.
We respectfully submit that this review misidentifies our contribution. The novelty of MuPHI is not the phenomenon of compositional harm, which we explicitly do not claim to have discovered, but a specific diagnostic question and the framework to answer it. The question we pose in the paper is, can a model identify the cross-modal mechanism responsible for implicit harm, and can that mechanism be evaluated beyond a binary label?
To answer Q2 directly : MuPHI reveals that VLMs systematically ground both modalities correctly yet fail specifically at cross-modal binding which is a failure mode that accuracy alone cannot detect and that no existing benchmark isolates. This finding changes how we interpret VLM performance on harm benchmarks i.e high accuracy does not imply compositional understanding, and improvements driven by dataset-specific correlations are indistinguishable from genuine reasoning gains without MuPHI’s controlled diagnostic design.**
We further address each concern in turn:
Concern: MuPHI is not novel relative to SIUO and ELITE (W1, Q2)
*We agree that the broader phenomenon of compositional harm is not new and we do not claim this. Our research question is more specific: can a model identify the cross-modal mechanism responsible for implicit harm, and can this be evaluated beyond a binary label? MuPHI is designed around this diagnostic question, with benign counterfactuals, harm-mechanism rationales, and multi-axis reasoning evaluation. Prior benchmarks (SIUO, ELITE) evaluate model behavior or output toxicity, neither of which requires isolating the compositional mechanism. The diagnostic framework MuPHI provides reveals that models ground both modalities correctly yet fail specifically at cross-modal binding, with harm-mechanism identifying the weakest dimension across all models and strategies (Figure 2). This finding requires the controlled conditions and rationale annotations that MuPHI introduces and could not have been established from existing benchmarks (SIUO, ELITE, FHM). We will sharpen this distinction in the introduction and related work.
Concern: Hateful Memes already contains counterfactual examples (W1, Q2)
That is correct, and our contribution is not the first use of counterfactual memes. In fact, we use FHM image counterfactuals as a complementary diagnostic. MuPHI adds a different controlled setting (not present in FHM), holding the image fixed while altering embedded text, together with harm-mechanism rationales and a five-axis analysis of model explanations. The novelty lies in how these pieces are combined to diagnose the failure of compositional interpretation, not in counterfactual pairing alone.
Concern: MuPHI mainly adds embedded text to existing settings (W1, Q2)
We respectfully disagree that embedding text within images is merely a superficial modification. Prior work, particularly [1], demonstrates that presenting text through the visual channel exposes distinct weaknesses in VLM safety alignment. Unlike SIUO and ELITE, which primarily investigate multimodal safety alignment and harmful model outputs, MuPHI evaluates whether VLMs can recognize and explain harmful meaning arising from image-text composition within a single visual input. This requires jointly interpreting visually embedded language, grounding it in the depicted scene, and identifying the resulting harm mechanism. Thus, our contribution is not simply embedding text into existing benchmarks, but introducing a mechanism-level diagnostic framework for a rather challenging setting.
[1] Gong, Yichen, et al. "Figstep: Jailbreaking large vision-language models via typographic visual prompts." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 22. 2025.