acceptodds
Withdrawn from ICLR 2027

MemeBench: Diagnosing Cultural Meme Interpretation with Rubrics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models have improved at describing visual content, but accurate descriptions do not ensure interpretation when meaning depends on knowledge beyond the pixels. Memes expose this gap because they rely on cultural entities, background knowledge, and community conventions. Most meme benchmarks reduce interpretation to labels or holistic scores, obscuring where an explanation breaks down. We introduce MemeBench, a diagnostic benchmark of 1,253 Chinese and English memes with human-written references and quality-controlled VIKR annotations, centered on anime, comics, games, and adjacent online subcultures. Its VIKR schema decomposes explanations into Visual clues, Identity links, Knowledge units, and Reasoning mechanisms, each scored against binary rubric criteria. Across 26 LVLMs, every model covers visible content more reliably than the knowledge needed to interpret it, and even the strongest retains a 22.6% Visual-Knowledge gap. To test whether this diagnosis can guide improvement, we introduce KAR, an entity-guided retrieval baseline built on CultureBase. Across four controlled models, KAR raises VIKR Success by 3.6-7.4% and, compared with generic retrieval, repairs more answers and breaks fewer. Yet both retrieval conditions improve Identity and Knowledge while reducing Visual coverage in every comparison. MemeBench thus turns meme interpretation from a single score into a rubric-level account of what a model sees, what it knows, and what retrieval repairs or breaks.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Multimodal Evaluation, Meme Understanding, Cultural Knowledge, Rubric-based Evaluation, Vision-Language Models, Retrieval Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_94QGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MemeBench, a bilingual Chinese-English benchmark for open-ended interpretation of cultural memes, comprising 1,253 samples. The authors propose the VIKR evaluation framework, which decomposes meme interpretation into four dimensions: Visual, Identity, Knowledge, and Reasoning. The framework evaluates the quality of model interpretations using fine-grained binary criteria. Based on this benchmark, the authors evaluate 26 large vision-language models (LVLMs), analyze the gap between visual description and cultural understanding, and propose KAR, an entity-guided retrieval method, to investigate how external knowledge can improve meme interpretation and what potential drawbacks it may introduce. The main contribution of the paper lies in advancing meme understanding from holistic correctness evaluation to dimension-level diagnosis, and in leveraging this diagnostic framework to analyze the benefits and trade-offs of retrieval augmentation.

Critical strengths
  1. Clear research problem. The paper addresses the gap between models' ability to describe images and their ability to understand their cultural meanings. Memes often rely on long-tail entities, background knowledge, and community-specific conventions, making them a suitable setting for investigating this capability gap.

  2. An interpretable evaluation framework. VIKR decomposes meme interpretations into four relatively distinct dimensions, helping identify whether a model fails to capture visual information, recognize entity identities, retrieve relevant background knowledge, or reason about the relationships among these elements. Compared with a single overall score, this approach provides more specific error diagnoses.

  3. Broad experimental coverage. The authors evaluate 26 commercial and open-source LVLMs and report overall performance, dimension-level results, and results on Chinese and English subsets. These evaluations enable readers to examine differences in model capabilities from multiple perspectives.

  4. Diagnostic value of retrieval-augmented experiments. The comparison between KAR and general-purpose web retrieval reveals an interesting phenomenon: retrieval can improve entity recognition and background knowledge coverage, but it may also reduce visual information coverage. The authors further analyze sample-level transitions between successful and unsuccessful interpretations rather than focusing solely on the final aggregate scores. This is a valuable aspect of the experimental design.

Critical weaknesses
  1. Dataset Quality. The authors describe MemeBench as a bilingual benchmark primarily focused on ACG and related internet subcultures, but its category distribution is imbalanced: ACG accounts for 50.1% of the samples, followed by cross-domain cultural references (21.2%), everyday life (13.2%), film and television (10.0%), and public life (5.4%). This distribution does not necessarily constitute a flaw in itself. However, given that cultural background understanding is the central motivation of the study, the authors do not sufficiently explain how the current category coverage supports the evaluation of broader meme-understanding capabilities, nor do they clearly justify the classification scheme. In addition, the Chinese and English subsets differ substantially in both size and source: the Chinese subset contains 768 samples, all collected from Bilibili, whereas the English subset contains 485 samples from Reddit and ImgFlip. The data collection process is also insufficiently described. The paper only states that the data were obtained from publicly accessible platforms and manually filtered, without specifying how meme images were collected from Bilibili, the formats in which the original content was available, or how candidate samples were retrieved and screened. Given that Bilibili is primarily known for its video content, this missing information makes it difficult to assess the traceability of the data sources and the reproducibility of the sampling process.

  2. Reliability of Data Annotation. All original reference interpretations and metadata in MemeBench were authored by a single annotator. Although the authors employ automated validation, targeted manual review, and independent verification on a stratified sample of 150 instances, these measures are insufficient to fully rule out potential biases introduced by single-annotator labeling. Such biases could further affect model evaluation results and subsequent diagnostic analyses.

Questions for authors
  1. Why is the domain distribution of the collected dataset so imbalanced, and what is the rationale behind this categorization scheme?

  2. Why is there such a large discrepancy between the numbers of Chinese and English memes? How exactly was the data collected?

  3. Were the annotators volunteers or paid annotators? Why were all metadata written by a single annotator, and how did you ensure that single-annotator labeling did not introduce subjective bias?

  4. Could you provide independent human verification results for the four dimensions and their binary scoring criteria on a representative subset of samples? Did the human review assess the correctness of the criteria themselves, rather than merely the semantic compatibility of the reference interpretations?

  5. Beyond SUCCESS, what is the agreement between the two LLM judges on each of the four dimensions—Visual, Identity, Knowledge, and Reasoning? If only one judge produces a different judgment, what are the main types of errors involved?

  6. When transitioning from CoT to Search-CoT, how do the authors control for the effects of changes in information extraction, search invocation, evidence formatting, and the reasoning workflow? Could additional ablation experiments more clearly disentangle the contribution of external evidence from that of the retrieval pipeline itself?

  7. Why does the English subset generally achieve higher experimental performance than the Chinese subset?

  8. Why is the VIKR framework designed as a sequential pipeline? Identity information can often be directly observed during the Visual stage, can't it?

  9. The dataset primarily consists of ACG-related memes, yet Figure 1 does not use an ACG example. Why is this the case, and what is the rationale for focusing primarily on ACG data in the first place?

Reviewer_XiDrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to study the gap between VLM ability to describe visual content of an image and VLM ability to interpret the visual content. The paper introduced MemeBench, a meme benchmark containing 768 Chinese and 485 English memes, focusing on anime, comics, games (ACG), and adjacent online subcultures. The annotations cover 4 angles: Visual clues, Identity links, Knowledge units, and Reasoning mechanisms (VIKR for short). Additionally, the paper proposed KAR, a retrieval-augmented generation pipeline involving a retrieval stage from a curated cultural entities database and an entity-based web search stage, as possible way to improve VLM ability to interpret the memes.

Critical strengths

Data source is well-motivated. Memes focused on anime, comics, and games domain likely require understanding of specific references to interpret correctly, thus allowing for a diagnosis of the gap between VLM ability to describe visual content of an image and VLM ability to interpret the visual content.

The VIKR annotations can facilitate diagnosis of a VLM’s failure modes in interpreting memes. Beyond the average VIKR and success scores, the paper also reports experimental results with granular distributions over the 16 VIKR pass/fail states for each VLM (Table 9), allowing for detailed comparison of different VLMs’ abilities.

Critical weaknesses

There is insufficient analysis about KAR. In Section 5.4, the author reports KAR repair-damage ratio is above 1 only for models that are already good at understanding memes. This should be discussed more extensively. Why are the possible reasons for this phenomenon? How are the repairs and damages distributed across VIKR measures? How can KAR be modified to help weaker models?

In Section 5.4, the paper claims that repair-damage ratio improves when using KAR over search-COT and COT for all 26 models, but paper did not report the retrieval experiment results beyond the Gemini-3.1-Pro, GPT-5.6-Sol, Kimi-K2.5, and Qwen3-VL-235B-A22B.

Questions for authors

As RAG pipeline seems to reduce Visual scores, have the authors considered adding visual clues and OCR from stage 1 as additional context for VLM answer generation in Stage 4?

Reviewer_nZvGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MemeBench, a benchmark for evaluating whether vision-language models can explain memes whose meaning depends on cultural knowledge. It contains 1,253 Chinese and English memes collected from Bilibili, Reddit and ImgFlip, centered on anime, comics, games and adjacent online subcultures. Each meme has a human-written reference explanation. An LLM converts each reference into a four-layer schema called VIKR (Visual clues, Identity links, Knowledge units and Reasoning mechanisms), with one to four binary rubric criteria per layer. Two LLM judges score a model's free-form explanation against these criteria, a criterion counts only when both judges agree, and an explanation succeeds only if every criterion passes. The authors calibrate this protocol against human labels, and they run a blind human study that validates the references and provides a human baseline. Across 26 models, the best model succeeds on about 60% of memes, whereas human explanations reach 83% on a subset. Every model covers visual content more reliably than background knowledge, and models with similar overall scores fail in different layers. The paper also proposes KAR, a retrieval baseline that grounds entities in a curated knowledge base before searching the web. On four models, KAR is compared with chain-of-thought prompting and with generic web search. Retrieval raises Identity and Knowledge coverage but lowers Visual coverage, and KAR improves success over generic search while repairing more answers and breaking fewer.

Critical strengths

S1. A well-motivated diagnostic design. The paper scores open-ended explanations against requirement-level rubrics in four layers, combining a strict all-layer success criterion with per-layer diagnosis. This shows where an explanation breaks down rather than only whether it is acceptable. The focus on long-tail entities and source works makes it hard to pass by knowing a few popular characters. The bilingual design and the case studies show that models with similar aggregate scores can fail in different places.

S2. Careful validation of the references and the automatic judge. Roles in the human study are separated: blind respondents, a reference author and an independent validator. This study checks reference compatibility and also provides a human baseline. The dual-judge intersection is calibrated against human labels and reaches agreement close to that between two human raters. The paper reports single-judge leaderboards and a relaxed-criterion sensitivity analysis. The authors release frozen responses and judge outputs, so every number can be recomputed.

Critical weaknesses

W1. Gemini wrote the rubrics, judges the responses and ranks first. Gemini-3.1-Pro converts every reference into rubric criteria (line 171), is one of the two judges (line 241), and tops the leaderboard, with Gemini-3-Flash second (Table 2). The GPT-only leaderboard (Table 8) rules out judge bias, but not rubric bias. The criteria are written in Gemini's words, and the paper describes only a "targeted" manual review of them (line 172). All references also come from a single annotator (line 167), and blind human explanations pass only 83.0%. Some failures may therefore reflect one specific reading rather than missing knowledge.

W2. KAR is under-specified, and its control does not isolate the evidence. The paper does not say how CultureBase was curated or how many benchmark entities it contains (line 1022). Line 295 says Appendix C gives the prompts, but the extractor and reasoner prompts are missing. Search-CoT changes the pipeline and the prompt as well as adding evidence (lines 312–313). The Visual drop attributed to "evidence crowding" (line 470) is therefore not isolated. KAR's gain over generic search is only 2.0–3.7 points.

Questions for authors
  1. Rubric authorship. If the rubrics for a subset are rewritten by another model or by hand, do the Gemini models' lead and the ranking still hold? Can you report judge–human agreement separately for each model family?

  2. KAR details. How was CultureBase built, and what fraction of the 2,072 benchmark entities does it cover? Can you add the missing prompts, and a control that runs the same pipeline with empty evidence?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.