acceptodds
Under review as a conference paper at ICLR 2027

Learning to Reason About Implicit Multimodal Harm: From Diagnosis to Generalization

Abstract

A benign image paired with a benign caption can become harmful only in combination, making detection fundamentally a cross-modal reasoning problem. Existing multimodal harm benchmarks evaluate only classification accuracy, leaving unclear whether models identify the cross-modal mechanism that produces harm. We introduce (ltimodal ragmatic arm nterpretation), a controlled diagnostic benchmark in which harm arises purely from image-text composition, with benign counterfactuals and annotated harm rationales. Using MuPHI, we systematically evaluate where compositional harm reasoning breaks in vision-language models (VLMs). MuPHI reveals that VLMs reliably ground each modality yet fail to bind the evidence into the harm mechanism and their predictions respond weakly to label-determining counterfactual interventions. Moreover, existing harm detection approaches trained on labels substantially improves in-domain (ID) detection but transfers poorly across harm distributions. We therefore propose ompositional arm nterpretation earning (), which rewards evidence identification in each modality, cross-modal binding and verdict consistency in the reasoning chain rather than just predicted label. CHIL improves 81% OOD transfer evaluations by an average of macro-F1 and correct counterfactual flip rate by up to and % points, while also improving overall reasoning quality over label-only optimization. These results suggest that optimizing how a model reasons, not just what it concludes, offers a promising direction for multimodal safety systems that generalize beyond dataset-specific correlations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Harm Detection, Compositional Reasoning, Diagnostic Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m6dMOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies implicit multimodal harm, where an image and text can be benign individually but harmful when combined. The authors introduce MuPHI, a benchmark designed to evaluate how well VLMs understand such harmful content. Their analysis shows that VLMs often struggle to connect information from images and text, even when they understand each separately. To address this, the authors propose CHIL, which trains VLMs to reason about harmful content rather than simply predict labels.

Critical strengths
  • Improving the cross-modal reasoning ability of VLMs is an important problem for multimodal safety and worth further investigation.

  • The analysis using counterfactual examples is useful. It helps identify whether VLMs actually understand how images and text interact to produce harmful meaning.

Critical weaknesses
  • My main concern is the limited novelty of MuPHI. The problem of benign images and text becoming harmful when combined has already been studied in VLM safety benchmarks such as SIUO [1] and ELITE [2]. MuPHI seems to mainly apply this idea to images with embedded text, but similar settings have already been explored in Hateful Memes [3], including counterfactual examples. Therefore, I find it difficult to see what is fundamentally new in MuPHI compared to existing benchmarks.

  • Another concern is the limited comparison with existing reasoning-based methods. Although the paper compares CHIL with SFT and outcome-only GRPO, it does not directly compare against recent methods [4]. Without these comparisons, it is difficult to assess whether CHIL provides meaningful improvements over existing approaches.

[1] Safe Inputs but Unsafe Output: Benchmarking Cross-Modal Safety Alignment in Vision-Language Models, NAACL 2025.

[2] Enhanced Language-Image Toxicity Evaluation for Safety, ICML 2025.

[3] The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes, NeurIPS 2020.

[4] ExPO-HM: Learning to Explain-then-Detect for Hateful Meme Detection, ICLR 2026.

Questions for authors
  • Could the authors compare CHIL with recent reasoning-based methods? This would help clarify the contribution of CHIL over existing approaches.

  • Regarding Weakness 1, could the authors clarify what new challenges MuPHI introduces compared to existing benchmarks?

4 comments

We respectfully submit that this review misidentifies our contribution. The novelty of MuPHI is not the phenomenon of compositional harm, which we explicitly do not claim to have discovered, but a specific diagnostic question and the framework to answer it. The question we pose in the paper is, can a model identify the cross-modal mechanism responsible for implicit harm, and can that mechanism be evaluated beyond a binary label?

To answer Q2 directly : MuPHI reveals that VLMs systematically ground both modalities correctly yet fail specifically at cross-modal binding which is a failure mode that accuracy alone cannot detect and that no existing benchmark isolates. This finding changes how we interpret VLM performance on harm benchmarks i.e high accuracy does not imply compositional understanding, and improvements driven by dataset-specific correlations are indistinguishable from genuine reasoning gains without MuPHI’s controlled diagnostic design.**

We further address each concern in turn:

Concern: MuPHI is not novel relative to SIUO and ELITE (W1, Q2)

*We agree that the broader phenomenon of compositional harm is not new and we do not claim this. Our research question is more specific: can a model identify the cross-modal mechanism responsible for implicit harm, and can this be evaluated beyond a binary label? MuPHI is designed around this diagnostic question, with benign counterfactuals, harm-mechanism rationales, and multi-axis reasoning evaluation. Prior benchmarks (SIUO, ELITE) evaluate model behavior or output toxicity, neither of which requires isolating the compositional mechanism. The diagnostic framework MuPHI provides reveals that models ground both modalities correctly yet fail specifically at cross-modal binding, with harm-mechanism identifying the weakest dimension across all models and strategies (Figure 2). This finding requires the controlled conditions and rationale annotations that MuPHI introduces and could not have been established from existing benchmarks (SIUO, ELITE, FHM). We will sharpen this distinction in the introduction and related work.

Concern: Hateful Memes already contains counterfactual examples (W1, Q2)

That is correct, and our contribution is not the first use of counterfactual memes. In fact, we use FHM image counterfactuals as a complementary diagnostic. MuPHI adds a different controlled setting (not present in FHM), holding the image fixed while altering embedded text, together with harm-mechanism rationales and a five-axis analysis of model explanations. The novelty lies in how these pieces are combined to diagnose the failure of compositional interpretation, not in counterfactual pairing alone.

Concern: MuPHI mainly adds embedded text to existing settings (W1, Q2)

We respectfully disagree that embedding text within images is merely a superficial modification. Prior work, particularly [1], demonstrates that presenting text through the visual channel exposes distinct weaknesses in VLM safety alignment. Unlike SIUO and ELITE, which primarily investigate multimodal safety alignment and harmful model outputs, MuPHI evaluates whether VLMs can recognize and explain harmful meaning arising from image-text composition within a single visual input. This requires jointly interpreting visually embedded language, grounding it in the depicted scene, and identifying the resulting harm mechanism. Thus, our contribution is not simply embedding text into existing benchmarks, but introducing a mechanism-level diagnostic framework for a rather challenging setting.

[1] Gong, Yichen, et al. "Figstep: Jailbreaking large vision-language models via typographic visual prompts." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 22. 2025.

Concern: CHIL lacks comparison to recent reasoning-based methods such as ExPO-HM (W2, Q1)

Table C:

MethodMuPHI (in-domain)FHMHarm-CHarm-POOD mean
ProCap89.649.441.034.941.8
RGCL90.256.520.136.637.7
SFT (structured prompt)91.658.766.054.159.6
ExPO-HM79.861.757.159.159.3
CHIL96.964.264.959.062.7

Comparison with ExPO-HM. We replicated ExPO-HM on MuPHI with the authors' released code and hyperparameters: policy-manual SFT, then curriculum GRPO with the CDE reward. We evaluated it with its own inference protocol on the same test items as all our tables. CHIL is better both in-domain (96.9 vs. 79.8 macro-F1) and on the unseen datasets (62.7 vs. 59.3 OOD mean).

Thank you for the detailed response. I appreciate the author's efforts to address my concerns.

While some of my concerns have been addressed, I still have concerns about the overall novelty of the paper. I understand the motivation behind this work and appreciate the effort to better understand implicit multimodal harm. However, I feel that the overall approach largely builds on existing research, and I am still not fully convinced that the paper offers sufficiently new insights or vulnerability.

Nevertheless, the additional clarifications and experimental results have addressed some of my concerns, so I am willing to raise my rating from 1 to 2.

We thank the reviewer for acknowledging our clarifications and for reconsidering their score. We have also included the clarifications in the updated paper (marked in blue in both main and supplementary) as suggested.

For additional clarity, we would like to further summarize the novelty and insights that MuPHI brings forth:

  • MuPHI is a controlled testbed (free from niche external references, unlike other harm datasets) built to investigate whether models identify the cross-modal mechanism that results in the harmful interpretation, and not just whether they can classify into a binary label.
  • Together with benign counterfactuals and human-annotated rationales in MuPHI, we provide insights that accuracy hides. VLMs ground and interpret each modality correctly but fail to bind them which yields the necessary harm interpretation.
Reviewer_SLkXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies compositional harms in multimodal (image+text) memes, where a benign image and a benign text become hateful/harmful when combined. The authors create a new controlled benchmark (MuPHI) with annotated harm rationales to diagnose the failure of zero-shot VLMs and training-based approaches; they also introduce a new training method (CHIL) based on GRPO with a lightweight LLM verifier that rewards evidence grounding and cross-modal reasoning. Experiments show that CHIL outperforms standard SFT and also improves transfer to OOD benchmarks.

Critical strengths
  • The paper presents a thorough diagnostic study of off-the-shelf VLMs on understanding compositional harms, exposing their common weakness in binding information from both modalities to predict the harm mechanism. The counterfactual samples enable further probing of the models' sensitivity to each input modality.
  • The training experiments uncover clear dataset bias among the existing benchmarks and severe overfitting from baseline methods (ProCap, RGCL), which produce even worse OOD results than naive SFT. The proposed methods set a simple and effective baseline in multimodal harm detection.
Critical weaknesses
  • While MuPHI appears to be a useful diagnostics tool, it too suffers from dataset bias and models trained on MuPHI (including those trained with RL) do not transfer well to OOD datasets. It would be helpful if the authors had gone one step further to analyze the sources of the distribution gap between these benchmarks, e.g. difference in annotation criteria, dataset-specific shortcuts, or visual/textual domain shifts.
  • I am not sure if the RL-SFT comparison is fair as SFT uses one-token prediction with no reasoning, while RL enables reasoning before the prediction. The additional reasoning verifier reward improves in-distribution performance vs. outcome-only RL, but does not seem to help OOD generalization (Table 13 last column). This again suggests the rationale/reasoning of harmfulness can be highly dataset-dependent, that the proposed training method is still exploiting the specific reasoning traces of training datasets, and that the primary OOD gain comes from RL rather than the proposed verifier reward.
Questions for authors
  • I wonder if the LLM verifier is susceptible to reward hacking during GRPO? Since it is text-only, could the model hallucinate visual evidence to maximize the verifier reward, causing visual grounding to degrade if trained for too long?
  • Have the authors ablated the SFT warmup? Since it uses a very different output format to that of RL (label-only vs. full reasoning trace), I am not sure if the warmup would be beneficial at all, since label-only SFT could suppress the output probabilities of all tokens except 0 and 1. Likewise, can this format mismatch make the KL penalty counterproductive too?
2 comments

We thank the reviewer for the constructive feedback and for appreciating the diagnostic analysis of failure modes that we intend to bring forward for harm detection mechanisms with MuPHI, alongside the importance of the counterfactuals and the simplicity of the method we propose. We further address the concerns regarding verifier and the warm start policy below. We believe the additional experimental results would help clarify the concerns.

Concern: MuPHI-trained models do not transfer well (W1)

  • We agree that training on MuPHI alone does not produce strong transfer and we argue this is a feature of the experimental design, not a limitation of the work.
  • MuPHI was primarily designed as a controlled diagnostic set, not as a replacement for the diversity of real-world training distributions.
  • We included MuPHI as a training source to test whether clean compositional examples help beyond their own distribution. The results show that compositional data alone is not enough, the reasoning supervision along with it is important for transfer.

Concern: CHIL may exploit dataset-specific reasoning traces (W1, W2)

We want to make this methodological distinction clear. CHIL does not supervise the policy by imitating dataset-specific gold or silver rationales. During GRPO, the frozen verifier scores the policy’s own generated explanation for cross-modal binding, alignment with its stated evidence, and consistency with its verdict. The verifier does not see reference rationales, so the proposed process reward is not directly fitting the wording of training-set explanations.

Concern: Could a text-only verifier reward hallucinated visual evidence? (Q1)

  • The verifier checks whether stated evidence is used consistently within the reasoning chain, it cannot verify faithfulness to the visual input, so we acknowledge this as a theoretical limitation.
  • However, Figure 4 shows VG remains near-saturated and stable across all training regimes, indicating CHIL does not in practice incentivize substituting hallucinated evidence for real visual content.
  • The independent rationale evaluation in Figure 4 provides a separate empirical check on visual grounding quality.

Concern: Why use label-only SFT as a warm start, and what about KL? (Q2)

  • Our approach follows the established SFT-to-RL paradigm [1,2], where SFT provides a task-adapted initialization and KL regularization prevents excessive policy drift during RL.
  • Specifically, label-only SFT establishes the harm-detection capability before GRPO introduces structured reasoning. The SFT checkpoint serves as both the initial policy and the fixed KL reference (β = 0.03), helping preserve task-specific knowledge while optimizing reasoning.
  • Additionally we run the ablation for CHIL with and without (starting from the base model) the warm-start SFT on Qwen2.5VL.
ModelMuPHI (in-domain)FHMHarm-CHarm-POOD mean
CHIL (SFT warm start)96.964.264.959.062.7
CHIL, no warm start77.160.256.951.856.3
Δ−19.8−4.0−8.0−7.2−6.4
  • Without the warm start, CHIL drops 19.8 macro-F1 points in-domain (96.9 → 77.1) and 6.4 points on the mean over the three unseen datasets (62.7 → 56.3), and it loses on every dataset.
  • The two stages are therefore complementary: SFT supplies the task prior, and CHIL's verifier-guided GRPO builds on it to improve cross-dataset transfer (59.6 → 62.7) and reasoning quality. We will add this ablation to the appendix.

[1] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in neural information processing systems 35 (2022): 27730-27744.

[2] Shao, Zhihong, et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models." arXiv preprint arXiv:2402.03300 (2024).

Concern: Verifier helps reasoning more than OOD generalization

We appreciate this observation and agree with the distinction. Our outcome-only ablation indicates that the optimization stage itself drives much of the transfer improvement. The additional verifier reward is better supported as a mechanism for improving the quality and internal consistency of explanations as we see in Figure 4. We will make this division of contributions clearer.

Dear Reviewer,

As the discussion period approaches its end, we wanted to politely follow up. We have addressed your main concerns above and have also uploaded a revised manuscript with changes marked in blue, both in the main text and supplementary.

We would greatly appreciate if you could take another look at our rebuttal and revised manuscript and reconsider your overall evaluation and score in light of our clarifications. Please let us know if any concerns remain, and we would be happy to provide further clarification.

Reviewer_366LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether vision-language models can recover the cross-modal mechanism by which an individually benign image and an individually benign caption become jointly harmful, and whether optimizing that mechanism rather than the label yields decisions that transfer across harm distributions. It offers (i) MuPHI, a 1,630-item diagnostic set (659 harmful / 971 benign) of generated images with overlaid captions, in which each harmful pair is paired with a same-image benign counterfactual and a silver rationale, plus a 330-item human-edited gold subset; (ii) a diagnosis of six open VLMs and GPT-5.5 under zero-shot, CoT and decomposition prompting, scored by GPT-5.4-mini on five reasoning axes alongside flip rate (FR) and correct flip rate (CFR), finding grounding saturated, harm-mechanism identification the weakest axis (52.1-76.0%), CFR at or below 35%, and label-only SFT strong in-domain (60-91 macro-F1) but 20-52 points worse out of distribution; and (iii) CHIL, a label-only LoRA warm start followed by GRPO whose reward adds a lambda-scaled, correctness-gated process term from a frozen text-only Qwen3-4B verifier that scores cross-modal binding, evidence alignment and verdict consistency. Across three backbones and four training sources CHIL is reported to beat its own warm start on 27/27 non-FHM transfer cells (mean +2.5 macro-F1) and to raise CFR by 0.4-5.4 points, with FHM as a training source reported as a net negative.

Critical strengths
  1. The FR/CFR pair is the paper's most transferable contribution and is what makes its claim checkable at all. Separating “did the intervention move the prediction” from “did it move in the right direction” turns an accuracy comparison into a mechanism test, and it yields one genuinely sharp result: aggregate macro-F1 and compositional sensitivity are decoupled, e.g. zero-shot decomposition reaches 75.6 macro-F1 on MuPHI while CFR stays at or below 29% (Fig. 3). The construction is reusable on any paired-counterfactual benchmark and is the part I would cite.
  2. The empirical surface is broad and, in several places, honestly reported against the thesis. Three backbones x four training sources x three seeds, a single prompt across datasets, full test splits, an evaluation judge deliberately different from the training verifier, and item-level accounting wherever the numbers go the wrong way: FHM as a training source costs -9.7 out-of-domain macro-F1 on Qwen2.5-VL, direct rationale SFT drops Harm-C to 38.3 against 74.2 for label-only SFT (Table 14), and zero-shot decomposition fixes two GPT-5.5 predictions while breaking five. Reporting the FHM failure instead of burying it is much of why the positive cells are credible.
Critical weaknesses
  1. Several cross-table comparisons appear to rest on settings that are not fully aligned, which makes the size of the reported gain difficult to pin down. Table 2's SFT column is labelled “evaluated on one-token prediction”, while A.4.3 suggests the SFT warm start was trained on a one-token target but GRPO and evaluation used the [GROUNDING]/[INFERENCE]/[VERDICT] prompt; A.4.2, meanwhile, describes all post-training evaluation as extracting the verdict from [VERDICT]. These may well refer to different stages, but the same configuration then reads 91.6 in Table 2 and 90.0 in Table 3, with per-cell differences between +14.0 and -3.0 - wider than a change of query format alone would suggest. Two points would benefit from clarification: whether CHIL, as the only model trained and queried in the same format, may partly reflect format adaptation rather than reasoning; and whether the exclusion of ProCap and RGCL in Section 5.2, justified by “Table 2 shows that SFT transfers better”, still holds if the tables are not strictly comparable. Statistically, with 96 test items and no confidence intervals, a seed spread of +-0.01 macro-F1, and +2.5 as the mean of nine positive cells (22.5/9, against +0.66 over twelve), 27/27 reads most naturally as a sign test. I would likewise value any human-agreement measurement for the Qwen3-4B verifier's three axes, given that its prompt supplies a cue vocabulary and grants gamma=0.25 partial credit on wrong answers; and Table 13's -1.1 / +0.1 / -1.5 for the verifier term, against +3.5 / +3.6 / -2.2 for outcome-only GRPO, leaves open whether the process term carries the transfer. My concern is about how much of the headline is verifiable as submitted, not about the direction of the work.
  2. The premise that harm arises from image-text composition is central to the design, and a direct measurement would strengthen it. No unimodal baseline is reported on MuPHI - neither an image-only nor a text-only classifier - and since each counterfactual pair holds the image fixed and changes only the caption, a model that ignored the image would still obtain a high correct flip rate, which makes CFR harder to read as evidence of cross-modal binding. Relatedly, the benign counterfactuals are model-generated positive or condemning phrases (A.1.2) against MPUP attack prompts, so a sentiment prior may account for part of the pattern; a neutral-paraphrase control would separate the two. Table 8 also makes the “no external knowledge” claim harder to sustain, since “strange food tastes” over chopsticks requires knowing the trope's valence. An image-only classifier, together with human accuracy and kappa on the 659/971 split, would let readers distinguish a compositional deficit from a caption prior combined with domain shift to synthetic images - both still plausible readings of the current results.
Questions for authors
  1. Could you clarify how the settings behind Tables 2 and 3 relate - which protocol each table's SFT numbers come from (prompt, decoding, checkpoint selection), and whether the 91.6 / 90.0 difference and per-cell differences of up to 14 points reflect the query format or different runs? If so, a format-only control (the same warm start with GRPO rewarded solely for emitting a parsable [VERDICT]) would show how much of the +2.5 is format adaptation. If the protocols are not comparable, does Section 5.2's argument still support excluding ProCap and RGCL, and could they be run under the Table 3 protocol on at least the Qwen2.5-VL backbone? Bootstrap confidence intervals at n=96, a note on whether +2.5 averages nine, twelve or thirty-six cells, any human-agreement measurement for the verifier's three axes, and your reading of Table 13's -1.1 / +0.1 / -1.5 against +3.5 / +3.6 / -2.2 would help. If the process term is not load-bearing, the restatement “GRPO on a structured rationale format transfers better than label-only SFT, while the verifier improves rationales without affecting accuracy” would also seem well supported.
  2. Would it be possible to report image-only and text-only classifiers on MuPHI, a check that each of the 659 generated images is individually benign without its caption, and human accuracy with inter-annotator kappa on the harmful/benign decision? Relatedly, does the CFR gain survive when the benign counterfactual is a neutral paraphrase rather than a positive phrase? These would help exclude the reading that the low flip rates partly reflect a caption-sentiment prior plus domain shift to synthetic images, which is hard to rule out given that an image-blind model would score highly on CFR by construction.
3 comments

We thank the reviewer for the constructive feedback and for identifying the FR/CFR pair as a reusable mechanism test and honest reporting of negative results as strengths. We address the verifiability concern directly, with a format control experiment (reported in Table A below) and clarifications on table inconsistency and +2.5 improvement (with bootstrap confidence intervals in Table B). We believe these together make the headline claim fully verifiable.

Concern: Confusion regarding Table 2 and 3 (W1, Q1)

  • We would like to clarify that Table 2 and Table 3 are measured under different protocols i.e only Table 2 reports one-token SFT evaluation, where we use the ‘SFT-only’ prompt described in Sec A.4.3. This makes the SFT evaluation in Table 2 directly comparable with exisitng works ProCap and RGCL.
  • However, Table 3 compares CHIL against its own SFT warm start under the standardized structured evaluation protocol with [GROUNDING]/[INFERENCE]/[VERDICT] extraction.

Concern: Regarding format adaptation (W1, Q1)

  • To separate learning the output format from learning to reason, we compare CHIL with three controls on Qwen2.5-VL-7B trained on MuPHI, all evaluated on identical test items (macro-F1; brackets: difference from CHIL with 95% paired bootstrap intervals):

Table A:

Test setSFT, one-tokenSFT, 3 tagsGRPO, [VERDICT] tag onlyCHIL
MuPHI (in-domain)95.888.589.596.9
FHM64.763.357.964.2
Harm-C67.965.661.764.9
Harm-P60.056.456.159.0
OOD mean64.261.858.662.7
Δ vs CHIL, in-domain−1.0 [−3.2, 0.0]−8.4 [−14.5, −3.2]−7.3 [−12.2, −2.8]–
Δ vs CHIL, OOD+1.5 [−0.7, +3.8]−0.9 [−3.1, +1.3]−4.1 [−6.3, −1.9]–

SFT, one-token: label-only SFT trained and queried in its native one-token format. SFT, 3 tags: SFT trained in CHIL's three-tag format, with the loss on the verdict only. GRPO, [VERDICT] tag only: GRPO from a label-only SFT warm start, rewarded for a parsable and correct [VERDICT] line.

  • The results show that format adaptation alone does not account for CHIL. GRPO rewarded only on the verdict tag is significantly below CHIL both in-domain (−7.3) and out of domain (−4.1). SFT trained in the same three-tag format recovers most of the transfer gap (−0.9 OOD, not significant) but remains 8.4 points below CHIL in-domain.

Concern: Decoding strategy, checkpoint selection (W1, Q1)

  • For all SFT runs, checkpoint selection was done on validation F1
  • We use greedy decoding for deterministic and reproducible evaluation with max_new_tokens = 1024
  • Further details about training and evaluation are reported in Appendix A.4.

Concern: Regarding contribution of verifier, improved OOD transfer and reasoning quality (W1, Q1).

  • CHIL improves over SFT on all 27 transfer cells across all backbones and training sources, with no single condition driving the result. We additionally computed bootstrap confidence intervals (1000 resamples) on the macro-F1 difference between CHIL and SFT on MuPHI.

Table B:

ModelVerifier effect (CHIL − outcome-only)Outcome-reward effect (outcome-only − SFT)
Qwen3-VL-8B−1.1 [−3.2, +1.0]+3.5 [+1.6, +5.6]
Qwen2.5-VL-7B+0.1 [−1.7, +1.7]+3.6 [+1.6, +5.5]
InternVL3-8B−1.5 [−3.2, +0.1]−2.2 [−4.0, −0.5]
Based on Table B results, that outcome-based GRPO accounts for much of the classification but the verifier improves the quality of the generated, especially cross-modal integration and harm-mechanism identification (Figure 4) and we agree with the restatement that you suggested and both contributions as meaningful for a system designed for content moderation. We will explicitly separate these contributions in the revised text.

Concern: Annotation reliability of MuPHI labels. (W2, Q2)

Standard Cohen’s kappa does not apply to our annotation protocol, which was a consensus-based filtering process rather than independent binary labeling. Any sample where either annotator expressed doubt was excluded rather than reconciled, with 52.9% of all candidate samples (images generated from harmful image prompt-text caption pairs in MPUP) dropped on these grounds. This is stricter than kappa-based schemes where disputed items are resolved by majority vote, since every included sample required consensus to pass. We will describe this protocol more explicitly in the revision.

Concern: Regarding unimodal baselines and CFR (W2, Q2)

  • CFR measures whether a model changes its prediction correctly under a controlled intervention. It is not, by itself, proof that the model used both modalities. Our intended claim is that CFR complements macro-F1 by exposing failures that accuracy alone conceals.
  • We further note that MuPHI includes gold unimodal annotations, which are human judgments of whether the image alone and the caption alone are harmful independently of their combination. Across all 659 harmful pairs, 85.9% of images and 81.1% of texts are judged individually benign by a consensus of human annotators, and the remaining as borderline/subjective in isolation (18.9% of texts and 14.1% of images). This indicates that, for a substantial proportion of examples, the harmfulness cannot be attributed to either modality alone but emerges from their combination. We acknowledge that neither metric independently proves multimodal grounding, but together, they provide a more informative evaluation of predictive performance and intervention sensitivity.

Concern: The claim that MuPHI requires no external knowledge is too strong (W2, Q2).

We agree that interpreting pragmatic implications requires general social knowledge. Our intended distinction is narrower. MuPHI is designed to reduce dependence on niche references to particular public figures, events or pop-culture artifacts that can complicate evaluation on memes sourced directly from the internet.

Dear Reviewer,

As the discussion period approaches its end, we wanted to politely follow up. We have addressed your main concerns above and have also uploaded a revised manuscript with changes marked in blue, both in the main text and supplementary.

We would greatly appreciate if you could take another look at our rebuttal and revised manuscript and reconsider your overall evaluation and score in light of our clarifications. Please let us know if any concerns remain, and we would be happy to provide further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

We thank all three reviewers for their detailed and constructive feedback.

Reviewers recognised the FR/CFR pair as a reusable mechanism test that decouples accuracy from compositional sensitivity, independent of this paper’s specific claims (366L). The paper’s honest treatment of negative results, including FHM as a training source, was noted positively (366L). The diagnostic finding that VLMs ground each modality correctly but fail to bind them was called thorough (SLkX), and the result that ProCap and RGCL overfit and transfer worse than plain SFT was highlighted as a valuable negative finding in itself (SLkX). The counterfactual analysis, using text flips to complement FHM’s image flips, was seen as strengthening the causal argument (m6dM).

Below we summarize the concerns raised, how each was addressed, and what changed in the revised manuscript (all changes are marked in blue):

  • Format-adaptation controls (366L, SLkX; App. A.5.2). Three controls on Qwen2.5-VL trained on MuPHI, with paired bootstrap confidence intervals (1,000 resamples): label-only SFT in its native format, SFT trained with reasoning tags, and GRPO with [VERDICT] tag. Verdict-only GRPO is below CHIL by 7.3 points in-domain and 4.1 points OOD. Format learning alone does not account for the gain.
  • SFT warm-start ablation (SLkX; App. A.5.3). Training from the base model, with everything else identical, drops 19.8 points macro-F1 in-- domain and 6.4 points OOD. SFT supplies the task prior, and CHIL's verifier-guided GRPO builds on it to improve cross-dataset transfer and reasoning quality.
  • Comparison with ExPO-HM (m6dM; App. A.5.4). CHIL outperforms ExPO-HM both in-domain (96.9 vs. 79.8) and OOD (62.7 vs. 59.3). The section also compares ProCap, RGCL, and SFT protocols on the same test items.
  • Unimodal annotation statistics (366L; Sec. 3). Across the 659 harmful pairs, 85.9% of images and 81.1% of texts are individually judged benign by annotator consensus (others flagged as borderline or subjective), confirming that harm in most pairs arises from the cross-modal combination rather than either modality alone.
  • Evaluation protocol (366L). All evaluation uses greedy decoding. GRPO checkpoints are selected by validation reward and SFT by validation macro-F1 (App. A.4.2). Tables 2 and 3 use different but intentional protocols. Table 2 evaluates SFT in one-token label-only format for direct comparability with ProCap and RGCL. Table 3 compares CHIL against its SFT warm-start under the structured [VERDICT] format. Both table captions now state this.
  • Statistical support (366L). CHIL improves all 27 non-FHM transfer cells (paired t-test, p < 10⁻⁴), with paired bootstrap intervals added for key comparisons.
  • Annotation protocol (366L). Consensus-based filtering excludes any sample where either annotator expressed doubt rather than reconciling disagreements, dropping 52.9% of candidates. This is stricter than majority-vote, which is why Cohen’s kappa does not directly apply (Sec. 3).
  • Verifier and RL design (366L, SLkX). Sec. 8 now separates the outcome reward and verifier contributions. The verifier is not supervised on dataset-specific rationales (Sec. 6) and visual grounding stays near-saturated across all training regimes (Fig. 4), confirming no drift toward hallucinated visual evidence.
  • Scope and metrics (366L, SLkX). MuPHI is a controlled diagnostic set (free from niche external references, unlike other harm datasets) designed to specifically analyze cross-modal information binding in VLMs for interpreting implicit harm. It is not meant as a replacement for diverse real-world training data (Sec 7). CFR as a metric exposes failures accuracy fails to reveal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.