SarcAtlas: Fine-Grained Annotation and Efficient Modeling of Multimodal Sarcasm

Multimodal sarcasm detection is commonly framed as binary classification. However, a correct prediction does not establish that the model understands why a sample is sarcastic. Meanwhile, annotation policy shifts across source datasets can induce dataset-specific decision criteria and limit cross-dataset generalization. We introduce SarcAtlas, a unified set of fine-grained annotations for MMSD, MMSD2.0, and the English and Chinese splits of SarcNet. SarcAtlas preserves each benchmark's original instances, splits, and labels. It adds annotations for strict and broad sarcasm, type and target, shared and modality-specific information, cross-modal relations, rationales, and interpretation dependencies. We control annotation quality through staged annotation, review, repair, and expert audit. Our analysis reveals systematic differences among benchmarks in label boundaries, pragmatic types, evidence structures, and interpretation requirements. Under the Original labels, cross-dataset evaluation mixes content shifts with annotation policy shifts. Based on these annotations, we develop a single-tower Qwen3-VL-Embedding model with five latent-token groups assigned to different annotated fields and a decision token. Training combines multi-task prediction, reconstruction of the annotated text fields, distillation of teacher-defined sample similarities, and label-supervised contrastive learning. At inference, a single image–text prefill supports binary detection and fine-grained prediction through parallel heads without autoregressive generation. The latent tokens are decoded only for case inspection. We evaluate in-domain detection, cross-dataset and cross-lingual transfer, case-level interpretability, and inference characteristics. Overall, SarcAtlas provides common decision boundaries for comparing multimodal sarcasm models, while its latent tokens make intermediate semantic information available for case-level inspection without sacrificing classification-based inference efficiency.