acceptodds
Under review as a conference paper at ICLR 2027

Multi-Quadrant Synthetic Data-driven Contrastive Learning for Hateful Meme Detection

Abstract

Internet memes are a common form of multimodal content. They usually express users' viewpoints through metaphors and may also carry implicit hateful messages, which makes hateful meme detection a challenging task. Current mainstream methods embed image and text features into a joint space and model their interactions; a meme is classified as hateful when its pattern matches known harmful features, such as a strong correlation between malicious text and inflammatory images. However, these methods ignore a key property of memes: two memes with highly similar surface content may convey completely opposite metaphors. Such confounding memes lie close to each other in the feature space but carry opposite labels, mixing together and making them hard to distinguish. To address this problem, we propose a Multi-Quadrant Synthetic data-driven Contrastive Learning (MQSCL) framework for hateful meme detection. The framework characterizes the decision space along two dimensions, surface similarity and metaphor consistency, and guides the decision boundary with metaphor consistency supervision. It consists of three core modules: (1) Metaphor Rationale Analysis: it uses a multi-agent pipeline that combines multimodal large language model interpretation with background knowledge retrieval to build a metaphor reasoning graph, locate the targeted group, and generate a metaphor rationale; (2) Quadrant Sample Generation: it takes the metaphor rationale as the condition and drives diffusion models with a text-image decoupled strategy, generating, for each source meme, four-quadrant counterpart samples that cover all combinations of surface similarity and metaphor consistency; (3) Multi-Quadrant Contrastive Learning: it applies regression, repulsion, and ordering losses to calibrate the similarities between the anchor and the quadrant samples, so that the decision boundary is determined by metaphor consistency rather than surface similarity. Quadrant samples are used only during training and introduce no additional inference cost. Experimental results show that MQSCL achieves better detection performance on mainstream datasets; on the most challenging FHM dataset, its F1 score is 3.9% higher than that of the state-of-the-art model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hateful Meme Detection, Synthetic Data, Multimodal Information Processing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2MPDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MQSCL for hateful meme detection, motivated by memes that share similar surface content but convey opposite hateful/non-hateful meanings. It organizes anchor–counterpart pairs along two dimensions: surface similarity and metaphor-polarity consistency. A knowledge-grounded rationale generation pipeline guides the synthesis of four quadrant counterparts for each training meme. A Quadrant Calibration Loss (QCL), comprising regression, repulsion, and ordering terms, calibrates their similarities to the anchor. The synthesis pipeline is used only during training data construction. Validation includes three benchmark datasets, ablations, comparisons across ten random seeds, human assessments of rationales and generated counterparts, and image-similarity analyses.

Critical strengths
  1. A clear and useful organization of synthetic training data. The four-quadrant formulation explicitly separates surface similarity from label consistency. Q2 and Q4 are particularly relevant because they address cases where similarity and class membership disagree. This provides a concrete framework for constructing complementary training examples around each anchor.

  2. Promising empirical results with practical inference requirements. The method improves performance across three benchmarks, and the ablations demonstrate benefits from both synthetic data augmentation and the complete QCL training scheme. The ten-seed comparison with re-run RGCL strengthens confidence in the reported improvements. Since retrieval and generation occur offline, the method introduces no additional inference-stage generation or retrieval.

Critical weaknesses
  1. The evidence does not fully establish the claimed connection between calibrated similarities and classification behavior, or the specific necessity of QCL. Section 3.3 applies classification supervision only to original anchors; generated counterparts are supervised through their similarities to the anchor. Separating opposite-label representations does not by itself ensure that the classifier assigns them different labels. Shared representations and anchor supervision can certainly make QCL useful, but Figure 3 demonstrates similarity changes rather than correct classification of the confounding counterparts. Consequently, the repeated claim that the decision boundary follows metaphor polarity is stronger than the evidence presented. Table 6 already includes a useful CE augmentation baseline. However, it replaces counterpart CE supervision with QCL rather than testing their combination. There is also no controlled comparison against standard supervised contrastive or triplet objectives on the same synthetic data. These omissions leave uncertainty about both the classification behavior induced by QCL and the contribution of its particular calibration structure.

  2. The generated Q4 counterparts are not sufficiently validated as examples requiring the cross-modal understanding emphasized by the motivation. Appendix D primarily preserves layout, subject type, visual style, text placement, and sentence style. These properties do not ensure that opposite-label counterparts remain difficult to distinguish through text alone. For example, Figure 3 changes “I beat children” to “I help children,” introducing an explicit lexical cue that may make the label distinction straightforward. The existing evaluations address related but different properties: Table 5 measures image–prompt alignment, Appendix E measures source–counterpart image similarity, and Appendix B states that human raters see the generated counterpart and its required polarity without the source meme. These checks support generation quality, but do not directly establish pairwise confusability or the need for image–text interaction. This matters because aggregate benchmark improvements could arise partly from learning direct textual cues, without demonstrating the proposed ability to resolve genuine multimodal confounders.

Questions for authors
  1. Clarification and classification evidence: Why does the complete method omit classification supervision on synthetic counterparts? Have you evaluated anchor CE + counterpart CE + QCL? Please also report the final classifier’s performance by quadrant, especially Q3/Q4, and the proportion of opposite-label pairs for which both members are classified correctly. Results on counterparts generated from held-out anchors would be particularly informative; training-counterpart accuracy would provide a diagnostic but would not establish generalization.

  2. Controlled comparison: On the same synthetic counterparts, with the same encoder, classification supervision, and comparable training budgets, how does QCL compare with standard supervised contrastive learning or a triplet loss? This would materially clarify whether the specific calibration targets and quadrant ordering offer benefits beyond general label-driven attraction and repulsion.

  3. Q4 validity: How well does a competitive text-only model perform when trained and evaluated on the corresponding synthetic data? Do you have results on an independent, manually verified set of real opposite-label confounding pairs, reporting the fraction of pairs with both predictions correct? Table 1’s text baseline and the current similarity evaluations do not directly answer these questions.

Reviewer_MfBAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MQSCL studies hateful meme detection using synthetic training counterparts organized by surface similarity and label consistency. A label-conditioned rationale pipeline retrieves background knowledge and guides the generation of four counterparts for each original training meme. The method trains a fusion classifier over frozen multimodal embeddings using classification supervision on the original memes and a Quadrant Calibration Loss on anchor–counterpart similarities. Experiments on FHM, Harm-C, and Harm-P report improvements over the included baselines, with ablations examining synthetic augmentation and individual loss terms.

Critical strengths
  1. The framework addresses the relevant ambiguity between surface similarity and hateful meaning.
  2. The reported benchmark results and component ablations indicate empirical benefits from synthetic augmentation and the proposed loss.
Critical weaknesses
  1. The contribution could be positioned more clearly relative to prior work on cross-modal semantic contrast [1], contrastive instruction tuning for confusable memes [2], and retrieval-augmented contrastive adaptation [3]. A focused discussion of how the four-quadrant construction and calibration objective extend or differ from these approaches would help establish the methodological advance.

  2. The experimental support for the supervision and its transferability could be strengthened. Counterpart ratings reveal the intended polarity to evaluators, and image-only similarity checks do not fully verify the image–text quadrant relations. Moreover, detector experiments use only frozen GME-2B features, leaving cross-backbone robustness uncertain. An independent audit would strengthen supervision evidence. A focused test with another encoder, or a clearer limitation of scope, would clarify transferability.

[1] Tian et al., Learning Multimodal Contrast with Cross-modal Memory and Reinforced Contrast Recognition, 2024.
[2] Hee et al., Contrastive Instruction Fine-Tuning Large Multimodal Model for Hateful Meme Classification, 2025.
[3] Mei et al., Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection, 2025.

Questions for authors
  1. Could you position the approach relative to cross-modal semantic contrast modeling [1], contrastive instruction tuning [2], and retrieval-augmented contrastive adaptation [3]? In particular, what additional modeling capability arises from jointly controlling surface similarity and polarity, and how does the calibration objective differ from existing label-aware representation learning? A conceptual comparison of the mechanisms and their intended capabilities would be sufficient.

  2. Could you supplement counterpart validation with an assessment that conceals the intended polarity and checks the source–counterpart surface relations? It would also be helpful to establish whether gains over a matched classification baseline persist with one different frozen encoder. A representative dataset and configuration would suffice; alternatively, an explicit discussion of backbone dependence would help delimit the conclusions.

Reviewer_AL7eOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper focusses on a core failure mode in multimodal hateful meme detection which is that a pairs of memes that can nearly be identical in surface content but convey opposite metaphors and thus should have opposite labels. The authors propose MQSCL which is a synthetic data-driven, contrastive learning framework that restructures the feature space to follow metaphor polarity rather than surface similarity.

They use (i) a multi-agent MLLM pipeline with retrieval to build a metaphor reasoning graph and produce a rationale, (ii) conditions a diffusion-based, text-image decoupled generator to synthesize four quadrant counterparts per meme along two axes surface similarity and polarity consistency, and (iii) optimizes a quadrant calibration loss that sets target similarities for polarity-consistent quadrants and upper bounds for polarity-opposite quadrants, with an ordering constraint.

Critical strengths
  • The quadrant calibration loss is a well-motivated composite objective encoding polarity before surface.
  • The text-image decoupled generation strategy (text-free diffusion + programmatic text overlay) is practically relevent.
  • The quadrant framework is intuitive and well-illustrated. The architecture and objective are specified clearly.
  • Avoids inference-time overhead by keeping counterparts as scaffolding for training only, making deployment more feasible.
  • Numbers have improved.
Critical weaknesses
  • This paper does not have the mandatory AI Usage Statement.
  • Reliance on proprietary, non-deterministic components (Gemini-3-Flash for rationale/prompting, FLUX.2 for generation, web retrieval) raises reproducibility concerns and makes replication challenging.
  • Results are not shown for datasets like MAMI. HARM-Covid and Politics perhaps have overlap in test and train samples from what I last remember. Did you do deduplication?
  • Recent strong benchmarks that reach SoTA (like See, Explain, Intervene) are not compared against. Further, since you are using encoder based approaches only, you must compare with the FHM winner model as well.
  • Generalizability is not shown. If I put FHM trained model to test on MAMI, how well does it generalize?
  • Your rationale and counterpart generation are conditioned on ground-truth labels. This is acceptable for training data construction, but the paper does not analyze robustness when the rationale agent errors or how noise propagates through QCL.
  • Sensitivity analysis is limited. No experiments are carried out by varying QCL margin/target values, the λ weight, or generator settings.
  • It is unclear that how general the gains are to different embedding backbones (e.g., CLIP, SigLIP, BLIP) and different fusion heads.
  • Synthetic-data artifacts could be exploited by the model (e.g., overlay font/layout/style, generator-specific textures); There is no control experiment that randomizes or harmonizes these to ensure the classifier is not shortcutting on generation signatures.
Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.