- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The Paper highlights the gap that exists in benchmarking the ability of multimodal LMs (in this case VLMs) to recognize implied meaning since implicit meanings in prompts are often misused for hate, toxicity or harmful behaviour by foundation models. The authors introduce MM-IMPBENCH, which is important to characterize the dimension missing from prior benchmarks, which only include task-specific labels. MM-IMPBENCH is built from the main contribution which is MM-IMPSCORE which scores the discrepancy between the surface-level meaning of an image-text pair and the real pragmatic meaning that is derived from the joint interpretation. The benchmark scores samples from six existing datasets which spans safety critical concepts. They find that detection accuracy of vision-language detectors reduces as implicitness increases, and moreover since difficulty is concentrated in the high implicitness regime, they direct these inputs through deep reasoning.
Critical strengths
I liked the following aspects of the paper, apart from the clear methodological design:
-
The two dimensional variation: explicitness and image dependence. Essentially there has been a blur of how indirect the instructions are versus how unrelated the image is to the text. Visual counterfactuals that keep the text and swap the image give a direct test of image sensitivity. Splitting by original image prevents a simple leakage route across family members.
-
In general, what was argued that is hidden by aggregate accuracy, is shown in the graded evaluation in my opinion. The downward progression of accuracy that is shown in the models from L1 to L4 levels of implicitness. They non trivially understand that broader capabilities while they do help with indirect questions, and moreover general-purpose strength need not mean the models can interpret implicit multimodal hate, offense and sarcasm.
Critical weaknesses
According to me, I have a few weaknesses/ gaps I found in the paper:
-
The score might be learning where the text comes from more than how indirect the pair is. In Figure 2 the I-D member keeps the original source text while the explicit members are LLM rewrites, so implicitness gets tied to natural posts versus generated prose. Moreover, the text-only branch recovers most of the ordering signal and the counterfactual accuracy is fairly low, so the score seems mostly textual even though the paper is motivated by image-induced shifts in meaning.
-
The decline from L1 to L4 is not separated from class balance or label quality. If positive items sit at higher scores, a detector that under-predicts positives would show the same drop without any real difficulty with indirectness. Essentially, without per-level label distributions, balanced accuracy, or human accuracy per level, part of the decline could come from noisier labels on the most indirect items rather than from the models.
Questions for authors
-
How does score-based routing compare with random routing at the same fraction of detailed-prompt calls? The latency savings follow from routing fewer items, so the real claim is that the score picks the right ones, and the accuracy margins over always-on reasoning are small.
-
Did the authors verify the implicitness levels through some form of human annotation on natural data? As in do humans believe L4 is highly implicit and the pragmatic meaning is truly hidden within layers of the surface level meaning, and is L1 direct?
5 comments
Response (Part II)
Q3: Controlling class composition and label quality in the level-wise decline.
A: Thank you for your valuable comment. Task-label quality control. Benchmark targets are inherited from peer-reviewed datasets with strict annotation protocols: FHM (Cohen's ), H-VLI (Fleiss' post-filtering), MMSD2.0 (), MultiOFF (Fleiss' 0.4–0.5), and SarcNet (). These reported agreements support the reliability of the ground-truth labels.
We fully agree that class composition must also be decoupled from implicitness when analyzing performance degradation. We therefore compiled per-level label distributions and computed balanced accuracy (BA), the macro-average of positive and negative recall, across all nine detectors.
Table C3: Class balance and detection performance by source. Entries follow L1 → L2 → L3 → L4 and are percentages.
| Dataset | Positive proportion | Accuracy | Balanced accuracy |
|---|---|---|---|
| FHM | 33.3 → 40.4 → 49.1 → 61.2 | 80.7 → 75.6 → 70.9 → 65.9 | 78.6 → 73.4 → 70.7 → 69.6 |
| H-VLI | 26.5 → 27.9 → 37.5 → 49.1 | 89.7 → 84.5 → 72.7 → 68.0 | 87.7 → 82.6 → 72.8 → 68.1 |
| MMSD2.0 | 27.7 → 43.0 → 48.2 → 54.6 | 78.2 → 74.4 → 73.4 → 72.2 | 74.9 → 74.1 → 73.4 → 72.2 |
| MultiOFF | 41.7 → 33.7 → 43.4 → 45.9 | 85.2 → 64.2 → 60.3 → 56.1 | 85.4 → 63.1 → 60.7 → 56.8 |
| SarcNet | 39.4 → 55.8 → 70.9 → 86.5 | 85.0 → 78.3 → 71.9 → 66.0 | 82.4 → 79.7 → 76.3 → 69.1 |
| ShieldVLM | 32.2 → 49.5 → 75.2 → 96.4 | 70.0 → 59.6 → 40.9 → 26.2 | 55.9 → 59.2 → 59.9 → 61.7 |
| Pooled benchmark | — | 79.1 → 73.3 → 68.3 → 65.4 | 72.4 → 70.7 → 68.2 → 65.7 |
Even after balancing the classes, our central finding remains robust: VLM detection performance still declines as implicitness increases. As shown in Table C3, pooled BA monotonically decreases from 72.4% to 65.7%. In ShieldVLM, detectors exhibit a strong conservative bias toward benign predictions. As L4 contains 96.4% toxic items, raw accuracy plummets to 26.2%, while BA slightly increases due to the extreme skew (only one benign example in L4). Examining class-specific recall confirms that moderation models frequently miss implicit toxicity when individually benign modalities interact.
Q4: Sample-selection gains at matched reasoning budgets.
A: Thank you for your insightful question. To prove that our routing gains stem from intelligent sample selection rather than simply increasing the count of expensive detailed-prompt calls, we evaluated MM-ImpScore routing against random selection under a strictly matched budget (i.e., identical counts of detailed-prompt calls per dataset). For the random baseline, we conducted 2,000 simulation runs.
Table C4 reports the simulated expected accuracy of random allocation alongside latency (which includes MM-ImpScore's scoring overhead under our manuscript's timing protocol).
Table C4: MM-ImpScore versus random routing at identical detailed-call counts.
| Detector | Scorer backbone | MM-ImpScore accuracy | Random accuracy | Gain (pp) | MM-ImpScore seconds/item | Random seconds/item |
|---|---|---|---|---|---|---|
| DeepSeek4.1-Flash | BLIP-2 | 73.5% | 71.3% | +2.20 | 3.94 | 3.74 |
| DeepSeek4.1-Flash | VLM | 73.5% | 71.1% | +2.43 | 3.54 | 3.35 |
| Qwen3.7-Flash | BLIP-2 | 73.7% | 72.6% | +1.15 | 16.90 | 16.47 |
| Qwen3.7-Flash | VLM | 73.9% | 72.4% | +1.48 | 14.81 | 14.62 |
Under random routing with fraction , the expected gain is simply times the improvement of detailed prompting on all items. MM-ImpScore exceeds this expectation by 1.15 to 2.43 percentage points at the same number of detailed calls, so the gain comes from selecting the right items rather than from reasoning more often.
Response (Part III)
Q5: Human verification of implicitness levels on natural benchmark data.
A: Thank you for your insightful question. Building upon our manuscript's human validation of synthetic families, we extended our human evaluation to natural benchmark data through two complementary protocols: absolute directness judgments and pairwise implicitness preference comparisons.
Q5 part 1: Absolute directness judgments
Three independent annotators evaluated 100 low-implicitness (L1) and 100 high-implicitness (L4) natural benchmark items presented in randomized order, judging whether each message was communicated directly or indirectly.
Table C5: Human directness judgments on 200 natural low- and high-implicitness examples.
| Measure | Result |
|---|---|
| Individual agreement: low-implicitness samples | 86.33% (259/300) |
| Individual agreement: high-implicitness samples | 81.00% (243/300) |
| Agreement of individual judgments with the assigned levels | 83.67% (502/600) |
| Agreement of majority judgments with the assigned levels | 86.50% (173/200) |
| Inter-annotator Fleiss' κ | 0.632 |
The results in Table C5 confirm strong alignment between human intuition and our assigned levels. MM-ImpBench's implicitness levels therefore agree well with how people actually perceive implicitness on natural data.
Q5 part 2: Pairwise implicitness preference on natural pairs
Annotators compared 100 same-source pairs, each containing one low- and one high-implicitness item, and chose the more implicit one or a tie. Baselines are direction-aligned so that a higher score predicts greater implicitness, with detector confidence from Qwen3.5-9B. Pair agreement tests whether the sign of a score difference matches the preference; Pearson r correlates the right-minus-left difference with preference coded −1/0/+1.
Table C6: Agreement with human preference on 100 natural benchmark pairs.
| Method | Pair agreement | Pearson r | Spearman ρ |
|---|---|---|---|
| MM-ImpScore | 80.00% | 0.6083 | 0.5741 |
| MM-ImpScore Text-only | 67.00% | 0.4215 | 0.4292 |
| ImpScore | 47.00% | −0.0879 | −0.0957 |
| DualCSE | 58.00% | 0.0972 | 0.0922 |
| CLIP inconsistency [2] | 56.00% | 0.1188 | 0.0888 |
| SLE lower simplicity [3] | 44.00% | −0.0850 | −0.1234 |
| Readabert reading difficulty [4] | 52.00% | −0.0104 | 0.0277 |
| ICNet image complexity [5] | 36.00% | −0.3197 | −0.3259 |
| Model Uncertainty | 65.00% | 0.3606 | 0.3356 |
As shown in Table C6, MM-ImpScore achieves the highest human agreement (80.00%) and demonstrates strong positive linear and rank correlations with human judgments. Notably, ICNet negatively correlates with human implicitness preferences (−0.3197 / −0.3259), directly refuting the assumption that generic visual complexity metrics can serve as surrogates for indirect intent.
Together, the two protocols show that on natural data MM-ImpScore follows human-perceived implicitness more closely than textual implicitness, image–text similarity, readability, visual complexity, or detector confidence, which makes it a sound basis for MM-ImpBench's evaluation axis.
References
[1] Barthes, R. (1977). Rhetoric of the Image. In Image–Music–Text, translated by Stephen Heath, 32–51. Fontana.
[2] Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML.
[3] Cripwell, L., Legrand, J., and Gardent, C. (2023). Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simplification. EMNLP.
[4] Naous, T. et al. (2024). ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment. EMNLP.
[5] Feng, T. et al. (2023). IC9600: A Benchmark Dataset for Automatic Image Complexity Assessment. IEEE TPAMI.
[6] Aggarwal, P. et al. (2024). Text or Image? What is More Important in Cross-Domain Generalization Capabilities of Hate Meme Detection Models? Findings of EACL, 104–117.
Response (Part I)
We are grateful for the recognition of the distinction between image dependence and implicitness, the value of visual counterfactuals, and the information revealed by graded evaluation. We address the questions on text provenance, visual sensitivity, class balance, routing, and human validation below.
In response, we have revised the manuscript to strengthen provenance and class-balance controls, expand visual-sensitivity and human-alignment evaluations, and include matched-budget random-routing comparisons.
Q1: Robustness to text provenance and rewriting style.
A: Thank you for your insightful question. We fully agree that performance shortcuts can arise if a scorer learns to distinguish natural user posts from AI-generated rewrites. Indeed, our preliminary analysis of textual ImpScore in Appendix C highlights this exact vulnerability: surface-form imbalances in training pairs cause models to overfit to trivial style edits, such as punctuation or emoji shifts (Table 12).
We therefore require style-preserving construction: rewrites imitate the source's punctuation, capitalization, register, and tone. To further reduce residual style correlations after this control, we apply training-time text augmentation, varying incidental surface forms while keeping the intended meaning fixed.
Table C1: Provenance control. Explicit/implicit ranking among regular family members, before and after removing the original text.
To test this directly, we removed the natural original-text member from every held-out test family. Explicit/implicit discrimination stays strong even when every compared item is AI-generated. Under the perturbation suite of Table 13 in our manuscript, augmentation reduces the change in the implicit–explicit score gap by about 89.6% under combined punctuation and emoji edits.
These results show that style-preserving construction and text augmentation substantially mitigate shortcuts arising from text provenance and rewriting artifacts.
Q2: The contribution of visual information to human-aligned implicitness measurement.
A: Thank you for your insightful question. Text often carries the primary interpretive signal in multimodal tasks, consistent with linguistic accounts of image–text communication [1], findings from cross-domain hateful-meme detection [6], and our gate-weight analysis (Table 16). This helps explain the modest accuracy drop when images are removed from ordinary, text-dominant families.
Table C2: Agreement with human ordering on 102 text-preserving counterfactual pairs.
As reflected in our counterfactual construction, images can act as context switchers in social-media communication, changing the intended meaning of otherwise identical text. Table C2 verifies this distinction on human-endorsed pairs: text-only methods cannot distinguish the two messages, whereas MM-ImpScore captures the image-conditioned change.
More importantly, human implicitness-ranking judgments on natural examples from MM-ImpBench reveal a larger difference between the full model and its text-only ablation, as shown in the following excerpt from Table C6 in Q5 part 2. The full model improves human-order agreement by 13.00 percentage points, with stronger Pearson and Spearman correlations.
Excerpt from Table C6: Agreement with human implicitness preferences on natural benchmark pairs.
Aggregate accuracy on controlled families, where counterfactual comparisons constitute a relatively small share, does not fully reveal this advantage: strong performance on text-dominant examples can obscure the importance of image–text interaction in natural communication.
The counterfactual controls and natural-data human evaluation show that MM-ImpScore’s multimodal formulation captures implicitness arising from image–text interaction in social-media messages, including image-conditioned shifts in meaning that unimodal scorers cannot distinguish.