acceptodds
Under review as a conference paper at ICLR 2027

MM-ImpBench: Measuring Multimodal Implicitness for Graded Evaluation of Vision–Language Models

Abstract

Vision–language models (VLMs) perform well across diverse multimodal tasks, while their ability to recognize implied meaning remains questionable, which is important for safety-critical tasks such as detecting coded hate and toxic language. Existing benchmarks largely characterize implicitness through phenomenon-specific categories or binary labels, providing little indication of how indirectly an image–text pair communicates its meaning or how model performance changes as such messages become more indirect. To capture this missing dimension, we argue that multimodal implicitness should be treated as a continuous, task-independent property that can be measured directly, rather than reduced to a phenomenon-specific label, and introduce MM-ImpScore, a reference-free metric for multimodal implicitness, grounded in the discrepancy between an image–text pair's surface compositional meaning and the pragmatic meaning that emerges from their joint interpretation. We train the model on a dataset of controlled semantic families that vary communicative directness while approximately preserving the intended proposition. Building on this measure, we introduce MM-ImpBench, which scores samples from six existing datasets spanning hate speech, offensive content, sarcasm, and multimodal safety and stratifies them into distinct implicitness levels. Evaluating vision–language detectors reveals a recurring decline in detection accuracy as indirectness increases. Since difficulty concentrates in the high-implicitness regime, we use MM-ImpScore to route only such inputs to detailed reasoning, matching or improving always-on reasoning at lower latency. We will release our code and data.

Primary area:
datasets and benchmarks
Keywords:
Multimodal implicitness, Vision-language model evaluation, Graded benchmarking, Multimodal content detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k2f8OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The Paper highlights the gap that exists in benchmarking the ability of multimodal LMs (in this case VLMs) to recognize implied meaning since implicit meanings in prompts are often misused for hate, toxicity or harmful behaviour by foundation models. The authors introduce MM-IMPBENCH, which is important to characterize the dimension missing from prior benchmarks, which only include task-specific labels. MM-IMPBENCH is built from the main contribution which is MM-IMPSCORE which scores the discrepancy between the surface-level meaning of an image-text pair and the real pragmatic meaning that is derived from the joint interpretation. The benchmark scores samples from six existing datasets which spans safety critical concepts. They find that detection accuracy of vision-language detectors reduces as implicitness increases, and moreover since difficulty is concentrated in the high implicitness regime, they direct these inputs through deep reasoning.

Critical strengths

I liked the following aspects of the paper, apart from the clear methodological design:

  1. The two dimensional variation: explicitness and image dependence. Essentially there has been a blur of how indirect the instructions are versus how unrelated the image is to the text. Visual counterfactuals that keep the text and swap the image give a direct test of image sensitivity. Splitting by original image prevents a simple leakage route across family members.

  2. In general, what was argued that is hidden by aggregate accuracy, is shown in the graded evaluation in my opinion. The downward progression of accuracy that is shown in the models from L1 to L4 levels of implicitness. They non trivially understand that broader capabilities while they do help with indirect questions, and moreover general-purpose strength need not mean the models can interpret implicit multimodal hate, offense and sarcasm.

Critical weaknesses

According to me, I have a few weaknesses/ gaps I found in the paper:

  1. The score might be learning where the text comes from more than how indirect the pair is. In Figure 2 the I-D member keeps the original source text while the explicit members are LLM rewrites, so implicitness gets tied to natural posts versus generated prose. Moreover, the text-only branch recovers most of the ordering signal and the counterfactual accuracy is fairly low, so the score seems mostly textual even though the paper is motivated by image-induced shifts in meaning.

  2. The decline from L1 to L4 is not separated from class balance or label quality. If positive items sit at higher scores, a detector that under-predicts positives would show the same drop without any real difficulty with indirectness. Essentially, without per-level label distributions, balanced accuracy, or human accuracy per level, part of the decline could come from noisier labels on the most indirect items rather than from the models.

Questions for authors
  1. How does score-based routing compare with random routing at the same fraction of detailed-prompt calls? The latency savings follow from routing fewer items, so the real claim is that the score picks the right ones, and the accuracy margins over always-on reasoning are small.

  2. Did the authors verify the implicitness levels through some form of human annotation on natural data? As in do humans believe L4 is highly implicit and the pragmatic meaning is truly hidden within layers of the surface level meaning, and is L1 direct?

5 comments
AuthorsOpenReview ↗

Response (Part I)

We are grateful for the recognition of the distinction between image dependence and implicitness, the value of visual counterfactuals, and the information revealed by graded evaluation. We address the questions on text provenance, visual sensitivity, class balance, routing, and human validation below.

In response, we have revised the manuscript to strengthen provenance and class-balance controls, expand visual-sensitivity and human-alignment evaluations, and include matched-budget random-routing comparisons.

Q1: Robustness to text provenance and rewriting style.

A: Thank you for your insightful question. We fully agree that performance shortcuts can arise if a scorer learns to distinguish natural user posts from AI-generated rewrites. Indeed, our preliminary analysis of textual ImpScore in Appendix C highlights this exact vulnerability: surface-form imbalances in training pairs cause models to overfit to trivial style edits, such as punctuation or emoji shifts (Table 12).

We therefore require style-preserving construction: rewrites imitate the source's punctuation, capitalization, register, and tone. To further reduce residual style correlations after this control, we apply training-time text augmentation, varying incidental surface forms while keeping the intended meaning fixed.

Table C1: Provenance control. Explicit/implicit ranking among regular family members, before and after removing the original text.

Comparison scopeComparisons per runBLIP-2 accuracyVLM accuracy
All regular members4,81891.41%93.88%
Original-text members removed2,40888.86%93.40%

To test this directly, we removed the natural original-text member from every held-out test family. Explicit/implicit discrimination stays strong even when every compared item is AI-generated. Under the perturbation suite of Table 13 in our manuscript, augmentation reduces the change in the implicit–explicit score gap by about 89.6% under combined punctuation and emoji edits.

These results show that style-preserving construction and text augmentation substantially mitigate shortcuts arising from text provenance and rewriting artifacts.

Q2: The contribution of visual information to human-aligned implicitness measurement.

A: Thank you for your insightful question. Text often carries the primary interpretive signal in multimodal tasks, consistent with linguistic accounts of image–text communication [1], findings from cross-domain hateful-meme detection [6], and our gate-weight analysis (Table 16). This helps explain the modest accuracy drop when images are removed from ordinary, text-dominant families.

Table C2: Agreement with human ordering on 102 text-preserving counterfactual pairs.

MethodHuman-order agreement
ImpScore50.00%
DualCSE50.00%
Text-only ablation50.00%
MM-ImpScore (BLIP-2)80.39%
MM-ImpScore (VLM)87.91%

As reflected in our counterfactual construction, images can act as context switchers in social-media communication, changing the intended meaning of otherwise identical text. Table C2 verifies this distinction on human-endorsed pairs: text-only methods cannot distinguish the two messages, whereas MM-ImpScore captures the image-conditioned change.

More importantly, human implicitness-ranking judgments on natural examples from MM-ImpBench reveal a larger difference between the full model and its text-only ablation, as shown in the following excerpt from Table C6 in Q5 part 2. The full model improves human-order agreement by 13.00 percentage points, with stronger Pearson and Spearman correlations.

Excerpt from Table C6: Agreement with human implicitness preferences on natural benchmark pairs.

MethodPair agreementPearson rSpearman ρ
MM-ImpScore80.00%0.60830.5741
MM-ImpScore Text-only67.00%0.42150.4292

Aggregate accuracy on controlled families, where counterfactual comparisons constitute a relatively small share, does not fully reveal this advantage: strong performance on text-dominant examples can obscure the importance of image–text interaction in natural communication.

The counterfactual controls and natural-data human evaluation show that MM-ImpScore’s multimodal formulation captures implicitness arising from image–text interaction in social-media messages, including image-conditioned shifts in meaning that unimodal scorers cannot distinguish.

AuthorsOpenReview ↗

Response (Part II)

Q3: Controlling class composition and label quality in the level-wise decline.

A: Thank you for your valuable comment. Task-label quality control. Benchmark targets are inherited from peer-reviewed datasets with strict annotation protocols: FHM (Cohen's κ=0.684\kappa = 0.684), H-VLI (Fleiss' κ=0.94\kappa = 0.94 post-filtering), MMSD2.0 (κ=0.811\kappa = 0.811), MultiOFF (Fleiss' κ\kappa 0.4–0.5), and SarcNet (κ=0.9769\kappa = 0.9769). These reported agreements support the reliability of the ground-truth labels.

We fully agree that class composition must also be decoupled from implicitness when analyzing performance degradation. We therefore compiled per-level label distributions and computed balanced accuracy (BA), the macro-average of positive and negative recall, across all nine detectors.

Table C3: Class balance and detection performance by source. Entries follow L1 → L2 → L3 → L4 and are percentages.

DatasetPositive proportionAccuracyBalanced accuracy
FHM33.3 → 40.4 → 49.1 → 61.280.7 → 75.6 → 70.9 → 65.978.6 → 73.4 → 70.7 → 69.6
H-VLI26.5 → 27.9 → 37.5 → 49.189.7 → 84.5 → 72.7 → 68.087.7 → 82.6 → 72.8 → 68.1
MMSD2.027.7 → 43.0 → 48.2 → 54.678.2 → 74.4 → 73.4 → 72.274.9 → 74.1 → 73.4 → 72.2
MultiOFF41.7 → 33.7 → 43.4 → 45.985.2 → 64.2 → 60.3 → 56.185.4 → 63.1 → 60.7 → 56.8
SarcNet39.4 → 55.8 → 70.9 → 86.585.0 → 78.3 → 71.9 → 66.082.4 → 79.7 → 76.3 → 69.1
ShieldVLM32.2 → 49.5 → 75.2 → 96.470.0 → 59.6 → 40.9 → 26.255.9 → 59.2 → 59.9 → 61.7
Pooled benchmark—79.1 → 73.3 → 68.3 → 65.472.4 → 70.7 → 68.2 → 65.7

Even after balancing the classes, our central finding remains robust: VLM detection performance still declines as implicitness increases. As shown in Table C3, pooled BA monotonically decreases from 72.4% to 65.7%. In ShieldVLM, detectors exhibit a strong conservative bias toward benign predictions. As L4 contains 96.4% toxic items, raw accuracy plummets to 26.2%, while BA slightly increases due to the extreme skew (only one benign example in L4). Examining class-specific recall confirms that moderation models frequently miss implicit toxicity when individually benign modalities interact.

Q4: Sample-selection gains at matched reasoning budgets.

A: Thank you for your insightful question. To prove that our routing gains stem from intelligent sample selection rather than simply increasing the count of expensive detailed-prompt calls, we evaluated MM-ImpScore routing against random selection under a strictly matched budget (i.e., identical counts of detailed-prompt calls per dataset). For the random baseline, we conducted 2,000 simulation runs.

Table C4 reports the simulated expected accuracy of random allocation alongside latency (which includes MM-ImpScore's scoring overhead under our manuscript's timing protocol).

Table C4: MM-ImpScore versus random routing at identical detailed-call counts.

DetectorScorer backboneMM-ImpScore accuracyRandom accuracyGain (pp)MM-ImpScore seconds/itemRandom seconds/item
DeepSeek4.1-FlashBLIP-273.5%71.3%+2.203.943.74
DeepSeek4.1-FlashVLM73.5%71.1%+2.433.543.35
Qwen3.7-FlashBLIP-273.7%72.6%+1.1516.9016.47
Qwen3.7-FlashVLM73.9%72.4%+1.4814.8114.62

Under random routing with fraction pp, the expected gain is simply pp times the improvement of detailed prompting on all items. MM-ImpScore exceeds this expectation by 1.15 to 2.43 percentage points at the same number of detailed calls, so the gain comes from selecting the right items rather than from reasoning more often.

AuthorsOpenReview ↗

Response (Part III)

Q5: Human verification of implicitness levels on natural benchmark data.

A: Thank you for your insightful question. Building upon our manuscript's human validation of synthetic families, we extended our human evaluation to natural benchmark data through two complementary protocols: absolute directness judgments and pairwise implicitness preference comparisons.

Q5 part 1: Absolute directness judgments

Three independent annotators evaluated 100 low-implicitness (L1) and 100 high-implicitness (L4) natural benchmark items presented in randomized order, judging whether each message was communicated directly or indirectly.

Table C5: Human directness judgments on 200 natural low- and high-implicitness examples.

MeasureResult
Individual agreement: low-implicitness samples86.33% (259/300)
Individual agreement: high-implicitness samples81.00% (243/300)
Agreement of individual judgments with the assigned levels83.67% (502/600)
Agreement of majority judgments with the assigned levels86.50% (173/200)
Inter-annotator Fleiss' κ0.632

The results in Table C5 confirm strong alignment between human intuition and our assigned levels. MM-ImpBench's implicitness levels therefore agree well with how people actually perceive implicitness on natural data.

Q5 part 2: Pairwise implicitness preference on natural pairs

Annotators compared 100 same-source pairs, each containing one low- and one high-implicitness item, and chose the more implicit one or a tie. Baselines are direction-aligned so that a higher score predicts greater implicitness, with detector confidence from Qwen3.5-9B. Pair agreement tests whether the sign of a score difference matches the preference; Pearson r correlates the right-minus-left difference with preference coded −1/0/+1.

Table C6: Agreement with human preference on 100 natural benchmark pairs.

MethodPair agreementPearson rSpearman ρ
MM-ImpScore80.00%0.60830.5741
MM-ImpScore Text-only67.00%0.42150.4292
ImpScore47.00%−0.0879−0.0957
DualCSE58.00%0.09720.0922
CLIP inconsistency [2]56.00%0.11880.0888
SLE lower simplicity [3]44.00%−0.0850−0.1234
Readabert reading difficulty [4]52.00%−0.01040.0277
ICNet image complexity [5]36.00%−0.3197−0.3259
Model Uncertainty65.00%0.36060.3356

As shown in Table C6, MM-ImpScore achieves the highest human agreement (80.00%) and demonstrates strong positive linear and rank correlations with human judgments. Notably, ICNet negatively correlates with human implicitness preferences (−0.3197 / −0.3259), directly refuting the assumption that generic visual complexity metrics can serve as surrogates for indirect intent.

Together, the two protocols show that on natural data MM-ImpScore follows human-perceived implicitness more closely than textual implicitness, image–text similarity, readability, visual complexity, or detector confidence, which makes it a sound basis for MM-ImpBench's evaluation axis.

References

[1] Barthes, R. (1977). Rhetoric of the Image. In Image–Music–Text, translated by Stephen Heath, 32–51. Fontana.

[2] Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML.

[3] Cripwell, L., Legrand, J., and Gardent, C. (2023). Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simplification. EMNLP.

[4] Naous, T. et al. (2024). ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment. EMNLP.

[5] Feng, T. et al. (2023). IC9600: A Benchmark Dataset for Automatic Image Complexity Assessment. IEEE TPAMI.

[6] Aggarwal, P. et al. (2024). Text or Image? What is More Important in Cross-Domain Generalization Capabilities of Hate Meme Detection Models? Findings of EACL, 104–117.

Reviewer_sJ2UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces MM-ImpBench, a benchmark designed to evaluate the ability of vision-language models (VLMs) to understand multimodal implicature at varying degrees of implicitness. The authors first propose MM-ImpScore, which quantifies the degree of implicitness in image-text samples by comparing the aligned cosine distance between compositional image-text representations and joint pragmatic representations. They then curate a graded benchmark comprising 4,136 samples selected from six existing multimodal datasets, dividing the samples into four levels of implicitness according to their scores. The authors further validate the scoring method through semantic-family ranking experiments, human annotations, image counterfactual experiments, and ablation studies. They also analyze the relationship between implicitness and task accuracy across nine vision-language models. In addition, the paper explores an implicitness-based reasoning-prompt routing strategy to reduce inference latency.

Critical strengths
  1. The research question is meaningful and has potential practical value. The paper attempts to transform multimodal implicitness from a qualitative linguistic phenomenon into a quantifiable and graded evaluation dimension. This provides a potentially useful framework for analyzing vision-language models' performance across different levels of interpretive difficulty. The experiments on inference routing based on the proposed scores also demonstrate potential efficiency gains. Therefore, the paper goes beyond merely introducing a new evaluation metric.
  2. The experimental framework is relatively comprehensive, and the scoring method is examined from multiple perspectives. The authors investigate the behavior of the scorer through semantic-family ranking, human annotation, image counterfactual analysis, and ablation studies. They further apply the method to multiple vision-language models. These experiments provide some evidence of the method's practical behavior and give the paper a relatively complete research structure.
Critical weaknesses
  1. The logical connection between the core scoring metric and the target construct is insufficiently established, leaving construct validity unproven. The paper interprets the aligned cosine distance between compositional representations and joint pragmatic representations as a measure of multimodal implicitness. However, this distance does not have an unambiguous or unique semantic interpretation. A larger distance may reflect image-text inconsistency, image dependence, representational biases, or other forms of semantic complexity, rather than a higher degree of implicitness. The existing human evaluations, counterfactual experiments, and ablation studies provide some supporting evidence, but they do not adequately distinguish between these competing explanations. Consequently, the central methodological assumption of the paper remains insufficiently validated.
  2. The novelty and comparative experiments are insufficient to establish the method's distinctive contribution. Selecting samples from existing datasets, constructing a graded benchmark using a scoring method, and analyzing how model accuracy changes with difficulty have practical value, but the novelty of these contributions is relatively limited. It is also unsurprising that models perform worse on more implicit or more difficult samples. The paper needs to demonstrate more convincingly that MM-ImpScore captures information that existing metrics cannot explain. For example, it should establish the independent contribution of MM-ImpScore compared with text-based implicitness metrics, multimodal semantic similarity, general-purpose sample difficulty estimation, or model confidence. The current evidence is insufficient to establish that the proposed scorer is either necessary for understanding multimodal implicitness or clearly superior to alternative methods.
Questions for authors
  1. The logical connection between the core scoring metric and the target construct is insufficiently established, leaving construct validity unproven. The paper interprets the aligned cosine distance between compositional representations and joint pragmatic representations as a measure of multimodal implicitness. However, this distance does not have an unambiguous or unique semantic interpretation. A larger distance may reflect image-text inconsistency, image dependence, representational biases, or other forms of semantic complexity, rather than a higher degree of implicitness. The existing human evaluations, counterfactual experiments, and ablation studies provide some supporting evidence, but they do not adequately distinguish between these competing explanations. Consequently, the central methodological assumption of the paper remains insufficiently validated.
  2. The novelty and comparative experiments are insufficient to establish the method's distinctive contribution. Selecting samples from existing datasets, constructing a graded benchmark using a scoring method, and analyzing how model accuracy changes with difficulty have practical value, but the novelty of these contributions is relatively limited. It is also unsurprising that models perform worse on more implicit or more difficult samples. The paper needs to demonstrate more convincingly that MM-ImpScore captures information that existing metrics cannot explain. For example, it should establish the independent contribution of MM-ImpScore compared with text-based implicitness metrics, multimodal semantic similarity, general-purpose sample difficulty estimation, or model confidence. The current evidence is insufficient to establish that the proposed scorer is either necessary for understanding multimodal implicitness or clearly superior to alternative methods.
6 comments
AuthorsOpenReview ↗

Response (Part I)

We are grateful for the recognition of a graded evaluation dimension and the breadth of our validation. The questions concern what MM-ImpScore measures and what it contributes beyond existing signals; we address them with controlled comparisons, human judgments, and graded model evaluation.

In response, we have revised the manuscript to include controlled semantic comparisons and human evaluations on constructed and natural data, comparing MM-ImpScore with related metrics to clarify what it measures.

Q1: Separating implicitness from image dependence, image–text inconsistency, and complexity.

A: Thank you for your insightful question. We validate implicitness against human judgments of how directly an image-text message communicates its intended meaning. Image dependence, inconsistency, and complexity can interact with implicitness, since image–text incongruity can itself contribute to irony, but they are distinct properties that we control for one at a time below.

Q1 part 1: Image dependence held fixed

On 200 randomly sampled test families, annotators verified the image-dependence labels. We then compared implicit and explicit members within the image-dependent condition and separately within the image-independent condition. Results are averaged across repeated model runs.

Table B1: Explicit/implicit discrimination at fixed image dependence. Score gaps are implicit minus explicit.

ComparisonPairsBLIP-2 pair accuracyBLIP-2 score gapVLM pair accuracyVLM score gap
Image-dependent: I-D versus E-D20094.83%+0.69795.50%+0.627
Image-independent: I-ID versus E-ID20089.17%+0.48593.25%+0.553

The model distinguishes explicitness within both conditions. Conversely, holding explicitness fixed, the image-dependent minus image-independent gap is +0.172/+0.070 for implicit members and −0.040/−0.003 for explicit members using BLIP-2/VLM. Implicitness and image dependence are therefore distinct properties, as we argue in Section 3: needing the image to supply overt content is not the same as conveying meaning beyond it.

Q1 part 2: Image–text inconsistency versus implicitness

To prove that MM-ImpScore measures pragmatic implicitness rather than surface-level image-text inconsistency (e.g., CLIP score), we conducted two diagnostic, paired intervention experiments (Table B2):

  1. COCO Caption Shuffling (1,000 pairs): Randomly shuffling image-caption pairs to disrupt visual-textual alignment. This introduces pure image-text mismatch without adding implicit or indirect meaning.

  2. Neutral Caption Replacement: 120 highly implicit hateful FHM/H-VLI test examples with relatively high original CLIP matching scores receive length-matched neutral visual descriptions generated by Qwen3.7-Plus and manually reviewed. Their images remain fixed while the external text changes from the original message to a literal caption.

Table B2: Experiments of image–text mismatching and neutral-caption replacement. Each cell reports before → after, followed by relative change.

Paired interventionMM-ImpScoreCLIP inconsistency, 1−cosine
COCO caption shuffling, 1,000 pairs0.9381 → 0.9851 (+5.01%)0.6928 → 0.8550 (+23.42%)
Original text → neutral caption, 120 pairs1.5961 → 0.8143 (−48.98%)0.7014 → 0.7312 (+4.25%)

Together, these two controls clearly isolate our construct from image-text inconsistency:

Experiment 1 demonstrates that MM-ImpScore does NOT simply measure image-text mismatch. When captions are randomly shuffled, image-text inconsistency increases drastically (CLIP inconsistency rises by +23.42%). However, because arbitrary mismatch does not create pragmatic implicitness, MM-ImpScore shows a much smaller change (+5.01%), proving it is not merely tracking alignment failure.

Experiment 2 demonstrates that MM-ImpScore specifically tracks implicitness, whereas CLIP inconsistency fails to reflect changes in indirect meaning. When an implicit hateful message is literalized into a neutral caption, its communicative indirectness is reduced. MM-ImpScore captures this shift with a sharp −48.98% drop. In contrast, CLIP inconsistency slightly increases (+4.25%), failing to reflect this reduction in implicitness.

CLIP inconsistency measures visual-textual mismatch but fails to track changes in implicitness, whereas MM-ImpScore isolates the pragmatic gap of implicit communication. Inconsistency and implicitness are therefore distinct properties: a pair can be arbitrarily mismatched while asserting only what it says, or closely matched while implying something else, so similarity cannot substitute for the axis.

AuthorsOpenReview ↗

Response (Part II)

Q1 part 3: Agreement with the human explicit/implicit grouping

On 187 binary-detection families evaluated by three annotators, we grouped the top-two human-ranked items as implicit and the bottom-two as explicit. We evaluated Top-2 overlap and cross-group pairwise ordering against various baselines.

The alternatives cover textual implicitness (ImpScore, DualCSE), image–text inconsistency (CLIP [1]), text simplicity/readability (SLE [2], Readabert [3]), image complexity (ICNet [4]), and detector uncertainty. Their directions are aligned so that a higher score predicts greater implicitness: CLIP uses one minus cosine similarity, SLE is negated, Readabert uses a probability-weighted reading-difficulty score over its predicted levels, and model uncertainty uses the negative log probability of the label selected by Qwen3.5-9B.

Table B3: Agreement with human explicit/implicit grouping on the same 187 families.

MethodTop-2 overlapCross-group pair agreement
MM-ImpScore83.87%84.96%
ImpScore69.30%70.32%
DualCSE69.52%70.68%
CLIP inconsistency60.65%62.83%
SLE lower simplicity47.77%47.62%
Readabert reading difficulty52.85%52.50%
ICNet image complexity50.00%50.00%
Model Uncertainty49.42%49.75%

MM-ImpScore substantially outperforms text-only implicitness, CLIP inconsistency, visual complexity (ICNet assigns tied scores because images are fixed within each family, yielding 50% under tie-aware scoring), and model uncertainty.

The axis is therefore human-grounded rather than metric-defined: the score achieves the strongest agreement with how people separate direct from indirect expression among the compared measures.

Q2: Independent information beyond textual implicitness, semantic similarity, difficulty, and confidence.

A: Thank you for your valuable comment. MM-ImpScore captures a distinct communicative dimension: how indirectly intended meaning is conveyed by the complete multimodal message. Alternative metrics target adjacent but fundamentally different phenomena: textual metrics miss visual triggers, similarity metrics (CLIP) measure alignment rather than intent, readability and visual-complexity metrics measure reading difficulty and image complexity, and uncertainty reflects internal model confusion.

Q2 part 1: Multimodal implicitness beyond textual scoring and semantic similarity

From the manuscript's MM-ImpScore user study, we use 102 counterfactual pairs unanimously endorsed by three annotators for both a meaning change and greater implicitness of the original. Their text is identical, so ImpScore, DualCSE, and our text-only ablation assign the same score to both members. Table B4 reports agreement with the human ordering; tied scores receive half credit.

Table B4: Agreement with human ordering on 102 text-preserving counterfactual pairs.

MethodHuman-order agreement
ImpScore50.00%
DualCSE50.00%
Text-only ablation50.00%
ICNet52.94%
MM-ImpScore (BLIP-2)80.39%
MM-ImpScore (VLM)87.91%

The additional visual sensitivity captures differences in communicated meaning beyond textual scoring and image complexity. When the text is held fixed, a text-only scorer cannot register an image-induced change in meaning and therefore cannot distinguish social-media messages in which different images give the same text different pragmatic meanings.

Multimodal semantic similarity also does not substitute for implicitness. As shown in Q1 (Table B2), caption shuffling substantially increases CLIP inconsistency but changes MM-ImpScore much less; replacing implicit messages with neutral captions sharply reduces MM-ImpScore while CLIP inconsistency slightly increases. These complementary responses show that MM-ImpScore captures communicative indirectness that image–text similarity alone does not quantify.

AuthorsOpenReview ↗

Response (Part III)

Q2 part 2: Natural benchmark pairs

We compare metric scores with human preferences on 100 same-source pairs from MM-ImpBench, each containing a low- and a high-implicitness example. Human choices allow ties, and all scores are oriented toward greater implicitness. We report pair agreement and Pearson r/Spearman ρ between right-minus-left score differences and human preferences coded −1/0/+1.

Table B5: Agreement with human preference on 100 natural benchmark pairs.

MethodPair agreementPearson rSpearman ρ
MM-ImpScore80.00%0.60830.5741
ImpScore47.00%−0.0879−0.0957
DualCSE58.00%0.09720.0922
CLIP inconsistency [1]56.00%0.11880.0888
SLE lower simplicity [2]44.00%−0.0850−0.1234
Readabert reading difficulty [3]52.00%−0.01040.0277
ICNet image complexity [4]36.00%−0.3197−0.3259
Model Uncertainty65.00%0.36060.3356

MM-ImpScore shows the strongest alignment with human intuition about implicitness on naturally occurring messages among the evaluated metrics. Together with the human grouping in Table B3, this shows that the axis measures what people perceive rather than what a proxy measures.

Q2 part 3: A new axis for model evaluation

MM-ImpBench translates MM-ImpScore into a unified evaluation axis across multimodal domains. While existing datasets (e.g., Latent Hatred, H-VLI, ShieldVLM) only provide discrete labels for whether content is implicit, MM-ImpScore continuously quantifies how indirectly it is expressed. This exposes hidden performance degradation and rank reversals obscured by overall accuracy.

As shown in Figure 5 of our manuscript, Qwen3-VL-8B and Qwen3.5-9B share near-identical overall accuracy but reverse rankings between L1 and L4. Similarly, Qwen3.7-Flash and Gemini-3.7-Flash match in overall performance yet diverge by 3.0 percentage points on L4. These fine-grained diagnostic profiles expose model-specific blind spots in indirect communication, offering actionable guidance for model selection and targeted alignment.

In summary, human validation, controlled interventions, and model diagnostic profiles collectively establish the necessity of this dimension. MM-ImpScore captures the gap between surface composition and pragmatic intent in strong alignment with human judgments of implicitness, measuring a dimension that existing metrics miss. This makes MM-ImpBench a reliable benchmark for diagnosing VLM performance under varying degrees of implicitness, helping identify weaknesses hidden by overall accuracy.

References

[1] Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML.

[2] Cripwell, L., Legrand, J., and Gardent, C. (2023). Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simplification. EMNLP.

[3] Naous, T. et al. (2024). ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment. EMNLP.

[4] Feng, T. et al. (2023). IC9600: A Benchmark Dataset for Automatic Image Complexity Assessment. IEEE TPAMI.

[5] Lin, T.-Y. et al. (2014). Microsoft COCO: Common Objects in Context. ECCV.

Reviewer_9wJXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studys how VLM behaves when given "implicit" image-text pairs. It first train a scoring model that rates the implicitness of the model. The model is trained on AI-rewritten implicit data as positive. Then this scoring model is used to divide existing hate or satire data into different levels, and the authors found the more implicit the data is, it is harder to classify them correctly as hate/satire. The evaluation was conducted on various datasets.

Critical strengths
  1. Characterizing the implicitness of image-text pairs is an interesting and important problem to study. People often communicate in implict and subtle language and those could be more difficult for an LLM or VLM to understand.

  2. The routing mechanism is potentially a practically useful method to reduce the latency/cost of VLMs. Although it may be more direct to rate the "difficulty" of the data too to gain such savings, instead of implicitness.

  3. The data range covered is this analysis is quite broad.

Critical weaknesses
  1. The idea is mostly an extension of the existing IMPScore developed in an LLM context, to a vision-language context, though there are new experiments and adaptations. The novelty of the work is therefore limited.

  2. The main contribution, the finding that more implicit image-text pairs lead to less accurate performance from VLMs is not surprising. It is to a large degree expected.

  3. The implictness scoring model is trained on AI-revised data pairs, not human expert annotation, which could be problematic. There is no hard guarantee that these are indeed more implicit rather than deviating from the original meaning completely. The model could just rewrite the text to potentially unrelated/tangent meaning of the image, and then naturally on those data VLMs would be less accurate.

  4. The work is a recategorization of existing data into different levels without new collection or annotation. Thus the contribution is more of an empirical analysis rather than benchmark project as the title suggests.

  5. The datasets used to train the scoring model and the final benchmark have some overlaps, which could make the findings less generalizable.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response (Part I)

We are grateful for the recognition of multimodal implicitness, the breadth of our evaluation, and the practical potential of implicitness-guided routing. We address the questions on novelty, data validity, benchmark construction, and generalization below.

In response, we have revised the manuscript to clarify the benchmark contribution, strengthen data-validity and generalization checks, and include semantic controls and matched-budget routing comparisons.

Q1: Multimodal reformulation of implicitness beyond textual scoring.

A: Thank you for your insightful question. Our contribution is a multimodal reformulation of implicitness that establishes a continuous, task-independent dimension for VLM evaluation.

The central modeling challenge is that image and text each convey meaning, while their interaction can evoke an additional, unstated proposition. Our Shared joint feature ablation represents a direct joint-feature extension of ImpScore to multimodal inputs: projecting the same fused representation into both branches leaves their semantic distinction entirely to the projections. We instead reformulate implicitness as the discrepancy between surface compositional and pragmatic meaning, using separate modality features for the former and joint features for the latter.

Furthermore, image and text can contribute differently across messages. Our adaptive modality gate learns their relative weights in the compositional branch, allowing surface meaning to reflect each pair's modality contributions rather than imposing equal weights.

Table A1 reports Intra accuracy on the held-out test set and agreement with three annotators' rankings on 187 binary-detection families from the 200-family human study (Section 5.3). The full model leads all three metrics. Together, semantic branch separation and adaptive modality weighting deliver a more accurate, human-aligned measure of multimodal implicitness than a direct joint-feature extension of Impscore.

Table A1: Held-out implicitness ranking and human agreement (%) for BLIP-2-based MM-ImpScore and its ablations.

MethodIntra acc.Family Top-2Family Pair
ImpScore64.0469.3070.32
Shared joint feature86.6778.6581.51
Equal modality weights89.9380.9383.16
MM-ImpScore90.5182.4084.71

We also mitigate the rewriting-style shortcut that textual supervision invites: Appendix C documents the surface-form imbalance in ImpScore's training pairs and its sensitivity to punctuation and emoji edits (Table 12), and our source-style-preserving construction with training-time augmentation mitigates it (Appendix C.1, Table 13). MM-ImpScore is thus a reference-free, task-agnostic measure of multimodal implicitness, which MM-ImpBench turns into a cross-task grading axis across hate, offensive-content, sarcasm, and safety detection.

Q2: Diagnostic value of a graded implicitness axis.

A: Thank you for your valuable comment. While it may seem intuitive that more implicit examples are harder to detect, prior work lacked a principled, continuous way to quantify how much and where models break down. Our contribution goes beyond confirming this trend: our fine-grained evaluation reveals model-specific weaknesses and ranking shifts, and identifies implicitness as a useful signal for reasoning allocation.

Existing benchmarks label whether an implicit phenomenon is present (e.g., Latent Hatred [4], H-VLI, ShieldVLM) but never how indirectly an image–text message communicates its meaning. We provide that missing axis: a continuous, reference-free score that profiles model behavior where discrete labels cannot.

For instance, as shown in Figure 5 of our manuscript (left table), Qwen3-VL-8B and Qwen3.5-9B exhibit nearly identical overall accuracy (differing by only 1.1 points), yet their rankings completely reverse between L1 and L4. Similarly, Qwen3.7-Flash and Gemini-3.7-Flash achieve comparable overall performance but diverge by 3.0 points on L4. These profiles locate recurring model weaknesses, guiding targeted interventions and resource allocation. Q6 demonstrates the value of implicitness as a routing signal motivated by these findings.

AuthorsOpenReview ↗

Response (Part II)

Q3: Implicitness and proposition preservation in AI-generated rewrites.

A: Thank you for your insightful question. The AI-rewritten families are used to train and validate MM-ImpScore, not to evaluate VLM detection. MM-ImpBench evaluates VLMs on original image–text samples and task labels from FHM, H-VLI, MMSD2.0, MultiOFF, SarcNet, and ShieldVLM, covering social-media hate, offensive content, sarcasm, and multimodal safety. These evaluation samples do not undergo our rewriting pipeline.

Table A2: Human verification of proposition preservation.

CheckSettingResult
Semantic preservation600 rewrites from 200 training families, 3 annotators, 1,800 judgments98.89%
Majority-vote preservation596 of 600 rewrites99.33%
Mean pairwise annotator agreementSame review98.44%

Table A3: Blinded implicitness judgments against construction labels and MM-ImpScore (reproduced from manuscript Table 2).

MeasureHuman–label agreementHuman–metric agreement
Top-2 hit86.25%82.00%
Cross implicit/explicit pair88.29%84.08%

We derive a semantic anchor from each source image–text pair, its task label, and available explanations to explicitly state the underlying proposition. This anchor guides rewriting to preserve that proposition while varying how directly it is expressed and whether interpretation requires the image (Appendices A.2–A.3). Three annotators assessed meaning preservation on 200 training families (Table A2) and blindly ranked implicitness on 200 held-out families (Table A3). Together, these evaluations show that the rewrites preserve the intended proposition while changing its implicitness, providing reliable training supervision that aligns the learned score with human judgments.

Q4: Benchmark contribution of regrading existing datasets.

A: Thank you for your valuable comment. As discussed in Q1 and Q2, MM-ImpBench combines a general-purpose implicitness scorer, graded evaluation data, and a shared protocol to provide a diagnostic axis beyond aggregate accuracy. Its level-wise profiles reveal model-specific weaknesses and ranking shifts, guiding targeted improvements and reasoning-resource allocation. MM-ImpScore is an integral component: its reference-free, task-independent scoring enables future extensions to new datasets and tasks. The current benchmark includes 4,136 source-aligned image–text examples spanning hate, offensive-content, sarcasm, and safety detection, with continuous scores and four shared implicitness levels.

AuthorsOpenReview ↗

Response (Part III)

Q5: Generalization under train–benchmark de-overlap.

A: Thank you for your insightful question. The training and evaluation collections draw on shared source datasets with limited implicit-category examples, particularly H-VLI and SarcNet, resulting in some shared items. We removed all 356 benchmark items present in scorer training and repeated evaluation on the remaining 3,780 items. Table A4 averages the nine detectors' accuracies, each computed from its valid outputs.

Table A4: Nine-detector mean accuracy after removing scorer-training overlap (%).

OverallL1L2L3L4L1-to-L4 decrease (pp)
71.0578.8673.39 ↓5.4769.53 ↓3.8667.42 ↓2.1111.44

The de-overlap results uphold our finding that VLM detection performance declines as implicitness increases. Mean accuracy decreases at every successive level, with an 11.44-point reduction from L1 to L4.

Q6: Implicitness versus generic difficulty as a routing signal.

A: Thank you for your valuable suggestion. We tested your suggestion by comparing MM-ImpScore with textual implicitness (ImpScore, DualCSE), text simplicity/readability (SLE [1], Readabert [2]), image complexity (ICNet [3]), and model uncertainty from Qwen3.5-9B. These measures prioritize, respectively, more implicit, less simple, harder-to-read, more visually complex, or less certain samples for detailed reasoning. Each method receives the same number of detailed-prompt calls within each dataset as MM-ImpScore.

Table A5: Alternative routing signals at matched detailed-call counts. Cells report accuracy (%) / mean seconds per item.

Routing methodDeepSeek4.1-FlashQwen3.7-Flash
Random71.1 / 3.3572.4 / 14.62
ImpScore70.8 / 3.3972.3 / 14.17
DualCSE71.1 / 3.4472.7 / 14.41
SLE lower simplicity70.4 / 3.3772.3 / 14.49
Readabert reading difficulty70.6 / 3.3172.7 / 14.78
ICNet image complexity70.6 / 3.4871.4 / 14.70
Model Uncertainty72.1 / 3.9272.5 / 16.10
MM-ImpScore73.5 / 3.5473.9 / 14.81

The matched-budget results show why implicitness is the right signal for allocating reasoning: in hate, sarcasm, and multimodal safety detection the difficulty lies in recovering an intended meaning the pair does not state, and MM-ImpScore models exactly that gap. Its routing accuracy exceeds general difficulty, readability, visual complexity, and answer confidence, at a scoring cost that is a small fraction of detection time, so it provides an efficient, task-independent way to direct additional computation toward messages that need deeper interpretation.

References

[1] Cripwell, L., Legrand, J., and Gardent, C. (2023). Simplicity Level Estimate (SLE): A Learned Reference-Less Metric for Sentence Simplification. EMNLP.

[2] Naous, T. et al. (2024). ReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment. EMNLP.

[3] Feng, T. et al. (2023). IC9600: A Benchmark Dataset for Automatic Image Complexity Assessment. IEEE TPAMI.

[4] ElSherief, M. et al. (2021). Latent Hatred: A Benchmark for Understanding Implicit Hate Speech. EMNLP.

Dear Reviewer,

With less than one day remaining in the discussion period, we would appreciate any feedback on our latest response. Thank you again for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback. We have made the following manuscript revisions, highlighted in pale yellow:

  • Clarified the benchmark contribution and diagnostic value of graded evaluation in Sections 2.1 and 6.1–6.2 (9wJX, sJ2U).
  • Added human verification of rewrite meaning preservation in Section 7.1 (9wJX).
  • Added counterfactual and image-dependence controls in Section 7.1, Tables 4–5 (sJ2U, k2f8).
  • Added human-alignment comparisons in Section 7.1, Tables 6–7 (sJ2U, k2f8), and further random-pair validation in Appendix D.6, Table 20 (sJ2U).
  • Added caption and direct-rewriting controls in Appendix D.4, Tables 17–19, and comparison-metric definitions in Appendix D.5 (sJ2U).
  • Added original-text provenance and class-balanced analyses in Appendices C.2 and E.4, Table 29 (k2f8).
  • Added evaluation excluding scorer-training overlap in Appendix E.4, Table 28 (9wJX).
  • Added matched-budget routing comparisons, including confidence-based self-routing, in Section 7.2, Table 8 (9wJX, sJ2U, k2f8).

We respond to each reviewer below and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.