- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MUNCH, a benchmark to test meme comprehension capabilities of multimodal large language models (MLLMs). The task is to match an image to its meme caption among four multiple-choice options. The benchmark curation depends on SemanticMemes, a large-scale dataset containing memes from different Reddit communities. The authors filter in total 123K high-quality memes, with the meme text removed by an off-the-shelf image in-painting model and automatically select hard distractor captions based on existing cluster annotations and the semantic similarity to the ground-truth caption. The benchmark contains three test partitions with different difficulty levels. The paper demonstrates that the tested closed and open MLLMs perform worse than humans by a large margin, and fine-tuning the models with a four-way contrastive objective can close the gap for the easy test split, but still can not catch up with the human performance when the semantics diverge from in-training data topics.
Critical strengths
- The paper deals with multimodal humor understanding, which still remains hard for existing frontier MLLMs, and the authors did report detailed analysis to analyze the shortcuts or the priors that may exist in the benchmark.
- The experimental evaluation involves different held-out test splits, which separate model performance based on known images, unknown images, and unknown semantic categories. Each split evaluates a distinct aspect of multimodal humor understanding, enabling a more systematic way to test the generalization capability of (fine-tuned) models.
Critical weaknesses
- The submission reads like a derivative work, an adaption of the cartoon image-text matching task proposed in (Hessel et al, 2023) to the memes from the SemanticMemes dataset (Zhou et al., 2024). The paper does not explain better why this problem is important and how it relates to/differs from previous work on computational humor in detail.
- The authors do not report the specific semantic clusters used in dev and test splits or the semantic distances between them. Measuring the overlap between training clusters and those in the Heldout Cluster (HC) partitions would help clarify the true difficulty of the test set. Without this, it is difficult to determine whether models are truly generalizing to semantically novel memes or simply interpolating between closely related concepts that were seen during training.
Questions for authors
- The authors include human evaluation according to moderately and highly familiarity to the memes, however this baseline is established with only two humans. The human evaluation should include a larger group. Also, it might be more useful to have two separate user groups, one with annotators who are highly familiar with memes and the one for moderately familiar to the memes. This way human baselines would be more meaningful.
- The analysis of the results is based on the final accuracy scores. To better understand the poor performances of the models, the thought traces of the models should also be inspected. What type of errors are common across different models, or model families?
- How did the authors decide the held-out templates and held-out clusters? Did they analyze the semantic distance between different clusters and the visual similarities of templates? Could the authors also analyze and report these distances and similarities to clarify the overlap between train, dev, and test splits?
- Similarly for the Held-out Template (HT), what is the overlap between the visual elements (objects, backgrounds, characters, etc.) in the held-out templates against those in the training set.
4 comments
- The authors include human evaluation according to moderately and highly familiarity to the memes, however this baseline is established with only two humans. The human evaluation should include a larger group. Also, it might be more useful to have two separate user groups, one with annotators who are highly familiar with memes and the one for moderately familiar to the memes.
The two annotators span moderate and high familiarity, and the gap between their scores (54.7% and 75.2%) is consistent with familiarity shaping human accuracy on MUNCH. The human performance estimate serves as a reference point, and the main conclusions compare models with one another and across partitions. We are actively trying to recruit additional annotators and to report moderate- and high-familiarity groups separately, as the reviewer suggests, and will report results as soon as able to (if completed before the rebuttal ends).
- The analysis of the results is based on the final accuracy scores. To better understand the poor performances of the models, the thought traces of the models should also be inspected. What type of errors are common across different models, or model families?
Appendix L analyzes which memes models answer wrongly, beyond aggregate accuracy. Few instances are missed by every model, and most instances are answered correctly by fewer than half of the models (Figure 13). A logistic regression of correctness on subreddit, template, caption length, and model places the most difficult memes in two groups (Figures 15 to 17). First, memes from topically niche communities (e.g., history, a single public figure, or the Marvel universe) presuppose community knowledge. Second, templates whose caption appears as a statement on a depicted screen or sign host semantically diverse captions. The reasoning-mode traces show one further error directly. The smaller Qwen3.5 models exhaust a 16K-token budget without reaching an answer on a quarter to half of KT responses (Table 8).
- How did the authors decide the held-out templates and held-out clusters? Did they analyze the semantic distance between different clusters and the visual similarities of templates? Could the authors also analyze and report these distances and similarities to clarify the overlap between train, dev, and test splits?
We chose held-out clusters and templates by random selection and instance counts, without reference to semantic or visual distance (Appendix D). HC clusters were drawn at random until the held-out clusters reached 7.5% of instances. HT took templates with fewer than 100 instances from multi-template clusters until HT reached ≈5% of instances. We report the semantic distances under the second weakness above and the visual overlap under Q4.
- Similarly for the Held-out Template (HT), what is the overlap between the visual elements (objects, backgrounds, characters, etc.) in the held-out templates against those in the training set.
Of the 179 HT templates, 93 (52%) share a named character, person, or franchise with at least one training template, and accuracy on HT is roughly the same regardless of whether the template shared an entity. The fine-tuned systems score higher only when the shared entity appears in a training template of the same cluster. Qwen3.8-27B tagged the named characters, people, franchises, and main objects in all 886 inpainted templates from the image, with no caption in the prompt As a baseline, 42% of training templates share an entity with another training template. A missed tag cause a template to go into the not-shared group, so 52% is a lower bound.
| System | Macro accuracy, shared entity (93) | Macro accuracy, no shared entity (86) | Difference [95% CI] |
|---|---|---|---|
| CLIP ViT-L/14-336, fine-tuned | 75.6 | 76.5 | −0.9 [−6.7, 5.0] |
| Gemma-4-E4B, LoRA | 58.0 | 56.9 | +1.1 [−7.1, 9.2] |
| CLIP ViT-L/14-336, zero-shot | 58.7 | 55.1 | +3.6 [−2.2, 9.9] |
| Gemini 3.5 Flash-Lite | 58.1 | 58.5 | −0.3 [−8.5, 8.0] |
Among HT templates, 45 share an entity with a training template in the same cluster. Fine-tuned CLIP ViT-L/14-336 scores 8.7 points higher on the 45 templates than on the other HT templates, and LoRA-tuned Gemma-4-E4B scores 17.1 points higher. The zero-shot systems score don't have an increased performance on these 45 templates (+0.4 points for zero-shot CLIP ViT-L/14-336 and −2.3 for Gemini 3.5 Flash-Lite).
For the image comparison, we matched each HT template to the most similar training template. By DINOv2 ViT-L/14, HT templates are more similar to the nearest training template than training templates are to each other (median cosine 0.773 against 0.63, p = 0.027), and CLIP ViT-L/14-336 shows no such difference (p = 0.28). Fine-tuned accuracy rises with DINOv2 similarity (Spearman's ρ = 0.27 for CLIP ViT-L/14-336 and 0.39 for Gemma-4-E4B). Zero-shot CLIP ViT-L/14-336 accuracy falls with DINOv2 similarity (ρ = −0.22), and Gemini 3.5 Flash-Lite shows no relation (ρ = −0.09). In §5 of the submission, the training template matched by image shares the instance's cluster for 48% of HT instances.
HC templates share entities with training templates at a similar rate (35 of 71), always with a template in another cluster, yet fine-tuned CLIP ViT-L/14-336 scores 62.2% on HC against 77.2% on HT, and LoRA-tuned Gemma-4-E4B scores 42.6% against 59.9%. We revised §4.2 and §6 to say that fine-tuned systems score highest on HT templates that resemble a training template of the same cluster, and we added the shared-entity counts and accuracies to Appendix D.1.
Thank you for your response and for running additional analysis. I found those results informative and appreciate the amount of work. That said, even with those additions, I still have reservations about the paper that it still lacks a detailed human study with expert analysis (Q1), and an extensive analysis of the reasoning failures (Q2). Hence, I will keep my overall score.
Thank you for the very detailed and thoughtful review. We've run the experiments and analyses you asked for and would love to hear your thoughts. To respond fully, we post our responses in a few comments.
MUNCH adopts the caption-matching format of Hessel et al. (2023) and applies the format to memes, whose humor depends on conventions a community attaches to a reused image. A New Yorker cartoon is drawn once for one contest, and the winning caption plays on the specific situation the cartoon depicts. In contrast, a meme template is reused across thousands of posts, and each caption fills the slot that relates to template's convention and meaning. A model therefore has to know (or learn) what a template is used for.
Because every template recurs, MUNCH holds out both (i) templates and (ii) entire semantic clusters at test time. The partitions test whether a learned convention transfers to a new template with a familiar meaning and to a template with an unfamiliar meaning (§3.3). Our results show that fine-tuning transfers to new templates in familiar clusters and largely fails to transfer to held-out clusters (§4 and §6). Held-out clusters have no counterpart in caption-contest data, where each cartoon appears once. In general, memes reflect the image–text humor that ordinary internet users produce and share, making them worthy of study as a humor medium.
HC clusters sit at the distance that separates neighboring training clusters, and models score no higher on the HC clusters closest to training. The submission reported only that the cluster IDs are disjoint, so we measured the distances. We represent each cluster by the mean Qwen3-Embedding-4B embedding of the cluster's captions, and each template by the CLIP ViT-L/14-336 and DINOv2 ViT-L/14 embeddings of the inpainted template image. For every HC cluster and template, we take the cosine similarity to the nearest training cluster or template and compare the similarity with the same measure between training clusters and between training templates.
We checked the caption comparison in two ways. First, centroids built from few captions have lower similarity to every other centroid, so we redrew every cluster at 30 captions (p = 0.07). Second, we removed the embedding component that all captions share by subtracting the mean training-caption embedding before forming centroids (p = 0.51). HC stays at the training baseline in both checks.
We then correlated each HC cluster's accuracy with the cluster's similarity to training, one value per cluster, for fine-tuned CLIP ViT-L/14-336, LoRA-tuned Gemma-4-E4B, zero-shot CLIP ViT-L/14-336, and Gemini 3.5 Flash-Lite. Spearman's ρ ranges from −0.34 to 0.42 across the four systems and three embeddings, and none were statistically significant.
By caption, HT templates are less similar to the other templates of the same cluster than training templates are though the magnitude difference is small (median cosine 0.96 vs. 0.97, p < 0.001). In a new Appendix D.1, we list the 21 HC clusters, each with instance and template counts, a representative template, a short description, and the nearest training cluster, and we report the distance comparisons, both checks, the accuracy correlations, and a figure of the similarity distributions.