Multi-Quadrant Synthetic Data-driven Contrastive Learning for Hateful Meme Detection

Internet memes are a common form of multimodal content. They usually express users' viewpoints through metaphors and may also carry implicit hateful messages, which makes hateful meme detection a challenging task. Current mainstream methods embed image and text features into a joint space and model their interactions; a meme is classified as hateful when its pattern matches known harmful features, such as a strong correlation between malicious text and inflammatory images. However, these methods ignore a key property of memes: two memes with highly similar surface content may convey completely opposite metaphors. Such confounding memes lie close to each other in the feature space but carry opposite labels, mixing together and making them hard to distinguish. To address this problem, we propose a Multi-Quadrant Synthetic data-driven Contrastive Learning (MQSCL) framework for hateful meme detection. The framework characterizes the decision space along two dimensions, surface similarity and metaphor consistency, and guides the decision boundary with metaphor consistency supervision. It consists of three core modules: (1) Metaphor Rationale Analysis: it uses a multi-agent pipeline that combines multimodal large language model interpretation with background knowledge retrieval to build a metaphor reasoning graph, locate the targeted group, and generate a metaphor rationale; (2) Quadrant Sample Generation: it takes the metaphor rationale as the condition and drives diffusion models with a text-image decoupled strategy, generating, for each source meme, four-quadrant counterpart samples that cover all combinations of surface similarity and metaphor consistency; (3) Multi-Quadrant Contrastive Learning: it applies regression, repulsion, and ordering losses to calibrate the similarities between the anchor and the quadrant samples, so that the decision boundary is determined by metaphor consistency rather than surface similarity. Quadrant samples are used only during training and introduce no additional inference cost. Experimental results show that MQSCL achieves better detection performance on mainstream datasets; on the most challenging FHM dataset, its F1 score is 3.9% higher than that of the state-of-the-art model.