acceptodds
Under review as a conference paper at ICLR 2027

SarcAtlas: Fine-Grained Annotation and Efficient Modeling of Multimodal Sarcasm

Abstract

Multimodal sarcasm detection is commonly framed as binary classification. However, a correct prediction does not establish that the model understands why a sample is sarcastic. Meanwhile, annotation policy shifts across source datasets can induce dataset-specific decision criteria and limit cross-dataset generalization. We introduce SarcAtlas, a unified set of fine-grained annotations for MMSD, MMSD2.0, and the English and Chinese splits of SarcNet. SarcAtlas preserves each benchmark's original instances, splits, and labels. It adds annotations for strict and broad sarcasm, type and target, shared and modality-specific information, cross-modal relations, rationales, and interpretation dependencies. We control annotation quality through staged annotation, review, repair, and expert audit. Our analysis reveals systematic differences among benchmarks in label boundaries, pragmatic types, evidence structures, and interpretation requirements. Under the Original labels, cross-dataset evaluation mixes content shifts with annotation policy shifts. Based on these annotations, we develop a single-tower Qwen3-VL-Embedding model with five latent-token groups assigned to different annotated fields and a decision token. Training combines multi-task prediction, reconstruction of the annotated text fields, distillation of teacher-defined sample similarities, and label-supervised contrastive learning. At inference, a single image–text prefill supports binary detection and fine-grained prediction through parallel heads without autoregressive generation. The latent tokens are decoded only for case inspection. We evaluate in-domain detection, cross-dataset and cross-lingual transfer, case-level interpretability, and inference characteristics. Overall, SarcAtlas provides common decision boundaries for comparing multimodal sarcasm models, while its latent tokens make intermediate semantic information available for case-level inspection without sacrificing classification-based inference efficiency.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Sarcasm, Dataset, Sentiment Analysis

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_tDTbOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SarcAtlas, a fine-grained multimodal sarcasm resource and a latent-token framework that uses structured annotations to jointly support sarcasm detection, attribute prediction, and semantic reconstruction.

Critical strengths
  • The authors conduct additional annotations that well align with the method’s motivation.
  • The framework combines efficient classification through a single multimodal forward pass with optional text reconstruction, allowing intermediate semantic representations to be inspected without requiring autoregressive generation for detection.
  • The paper is well written and presents solid work.
Critical weaknesses
  • The annotations are produced through an almost entirely automated pipeline, with only the 2.4% of samples that failed to pass automated review undergoing expert verification. Manual checks for cases that passed ROUND 1 and 2 seemed to be missing to ensure the automated annotation pipeline is reliable.

  • The validation of the semantic information captured by the latent-token groups is limited. While semantic reconstruction appears reasonable, the rationale of design is not sufficiently explained, particularly the use of different models (Qwen3-VL-Embedding-2B & Qwen3-0.6B) for representation extraction and text decoding, and the choice to use embedding-based predictions rather than direct decoding despite adopting a generative backbone.

  • A minor point regarding illustration: Figure 3 provides a detailed breakdown of the training components, but a clearer overview of the complete model architecture would make the method easier to follow. eg: The Figure highlights LoRA, but does not clearly identifying the other trainable components, like prefill tokens and the projector, which can be a little bit confusing for readers at first. This is a minor presentation issue that affects readability rather than a fundamental weakness of the approach.

Questions for authors
  • Figure 1 presents four categories. Does category D, “Noise,” refer to incorrect annotations in the original dataset? However, the annotation schema in Section 2 states that the Original label is retained. Are the labels verified and corrected during annotation? What is the specific procedure?
  • Why is Qwen3-VL-Embedding-2B used for representation extraction, while Qwen3-0.6B is used for reconstruction? I think the reason should be more clearly specified in the Method section.
5 comments

Dear Reviewer tDTb

We express our heartfelt gratitude to the reviewer for the highly positive evaluation, generous encouragement, and constructive guidance.

We sincerely appreciate your recognition of our fine-grained annotation effort, our single-forward efficiency, and our decoupled text reconstruction design. Your positive assessment that our paper presents well-written and solid work provides tremendous motivation for our team.

Your insightful questions regarding automated pipeline audits, legacy dataset noise, and illustration clarity have provided valuable direction for refining our paper. We have conducted extensive supplementary experiments and prepared detailed clarifications to address every point thoroughly. We present our systematic responses and empirical data below. Thank you again for your time, support, and dedication.

Q1

Weakness 1 The annotations are produced through an almost entirely automated pipeline, with only the 2.4% of samples that failed to pass automated review undergoing expert verification. Manual checks for cases that passed ROUND 1 and 2 seemed to be missing to ensure the automated annotation pipeline is reliable.
Question 1 Figure 1 shows four categories. Does Category D "Noise" refer to mislabeling in the original datasets? However, the annotation scheme in Section 2 states that original labels are preserved. Are labels verified and corrected during annotation, and what is the specific process?

A1

Thank you for your thorough review and constructive questions. Below, we report our human verification experiments, the definition of Category D in Figure 1, and our handling of original labels.

To verify samples that passed automated review, we conducted a quantitative audit and will add it to Appendix B.1. We drew a stratified random sample of N=500N = 500 records from the full 41,812 deduplicated records across all four sources. Two independent experts with backgrounds in natural language processing re-annotated these 500 samples under a double-blind protocol without seeing original labels or model outputs. The agreement metrics are summarized in the table below.

Evaluated FieldSample Size (NN)Inter-Annotator Agreement (IAA Kappa)Human-Model Accuracy (Acc, %)Human-Model Macro-F1 (%)Human-Model Agreement (Cohen's Kappa)
Strict Sarcasm5000.906897.6095.440.9089
Broad Sarcasm5000.955196.8095.730.9147
Text Sarcasm5000.911397.0092.740.8550
Image Sarcasm5000.809797.4081.760.6369
Evidence Source5000.880994.6079.140.8712
Pragmatic Type5000.885090.2040.010.8643

Across all fields, IAA Kappa ranges between 0.8097 and 0.9551, indicating almost perfect agreement. On primary tasks, human-model Cohen's Kappa reaches 0.9089 for Strict and 0.9147 for Broad sarcasm with 97.60% and 96.80% accuracy. These numbers confirm that our multi-stage reflection and verification pipeline produces reliable annotations.

Regarding Category D "Noise" in Figure 1, Category D represents data noise and pseudo-positives from legacy datasets, with a prominent subcategory being "Mention, not use". Original datasets were collected using keyword and hashtag queries (such as searching #sarcasm). This collection method collected posts that contain the word "sarcasm" or meme templates without expressing actual sarcasm. Examples include personality memes listing personal traits (Table B6 Case d) or posts discussing the concept of sarcasm without using it (Figure 1 Case D). These instances contain no irony, attitude reversal, or mockery. Legacy datasets mislabeled them as positive, creating keyword shortcuts for models. In SarcAtlas, we label the pragmatic type of this subcategory as Sarcasm mention and assign both Strict and Broad ground truths to non-sarcasm (Neither), removing these false positives.

Regarding the statement in Section 2 about preserving original labels, this refers to preserving metadata for historical reference. We retain original binary label fields in dataset records strictly for historical benchmarking and auditing. Our annotation workflow operates as follows. Tool-augmented multimodal models generate initial structured annotations. Independent multimodal models conduct logical verification. Non-conforming samples undergo targeted repair, and difficult edge cases (2.4%) are sent to human experts. Together with our 500-sample double-blind audit, this forms a closed quality-control process. We will clarify this wording in Section 2 and Figure 1 of the revised manuscript.

Q2

Weakness 2 The validation of the semantic information captured by the latent-token groups is limited. While semantic reconstruction appears reasonable, the rationale of design is not sufficiently explained, particularly the use of different models (Qwen3-VL-Embedding-2B & Qwen3-0.6B) for representation extraction and text decoding, and the choice to use embedding-based predictions rather than direct decoding despite adopting a generative backbone.

Question 2 Why is Qwen3-VL-Embedding-2B used for representation extraction, while Qwen3-0.6B is used for reconstruction?

A2

Thank you for this question. Our design choices are guided by two core factors, decision reliability and inference efficiency.

First, evaluating decision reliability shows that requiring models to generate explanations before predicting labels introduces narrative hallucinations and rationalization bias. As shown in the table below, Qwen3-VL-32B achieves 67.26 Strict F1 in direct classification, but its Strict F1 drops to 56.19 in explain-then-classify mode (-20.83% accuracy). Similarly, generative Qwen3-VL-2B reaches only 54.87 / 54.29 Strict F1, and generative fine-tuning (Native SFT-2B with 74.50 / 74.29) lags behind SarcAtlas. Latent-token single-forward classification isolates the decision manifold from free-text generation errors.

Model / ArchitectureInference ModeStrict Macro-F1 (%)Broad Macro-F1 (%)Strict Accuracy (%)Broad Accuracy (%)
Qwen3-VL-2B-InstructDirect classification54.8749.6765.9050.26
Qwen3-VL-2B-InstructExplain then classify54.2958.6655.7659.12
Qwen3-VL-4B-InstructDirect classification51.3854.7373.2266.92
Qwen3-VL-4B-InstructExplain then classify57.1462.6958.3663.24
Qwen3-VL-8B-InstructDirect classification61.4062.0673.0969.58
Qwen3-VL-8B-InstructExplain then classify61.7666.7163.0367.06
Qwen3-VL-32B-InstructDirect classification67.2674.7277.2577.56
Qwen3-VL-32B-InstructExplain then classify56.1963.6456.4263.95
Native SFT-2BDirect classification74.5074.2978.8576.53
SarcAtlas-2B (Ours)Single forward pass77.9279.5983.3880.63

Second, evaluating engineering efficiency on NVIDIA H200 GPUs demonstrates that single-forward classification provides major throughput gains. Modern GPUs achieve high compute utilization during dense prefill operations (SarcAtlas achieves 46.97% MFU). In contrast, autoregressive generation is limited by memory bandwidth. As shown in the table below, SarcAtlas processes 35.39 samples per second with a latency of 113.02 ms per batch. When generative models produce explanations, throughput drops to 1.34 to 4.81 samples per second and latency rises to 830 to 2979 ms. SarcAtlas delivers a 7.3x to 26.4x throughput advantage.

Method / Inference ModeOutput Content and FormatThroughput (samples/s) ↑Latency (ms/batch) ↓
SarcAtlas-2B (Ours)Deterministic labels + all fine-grained attributes35.39113.02
Qwen3-VL-2B (Generative)Label only30.54130.99
Qwen3-VL-2B (Generative)32-token short explanation + label4.81830.82
Qwen3-VL-2B (Generative)64-token medium explanation + label2.601536.32
Qwen3-VL-2B (Generative)128-token detailed explanation + label1.342979.81

Regarding model selection, decoupling Qwen3-VL-Embedding-2B from a frozen Qwen3-0.6B decoder protects embedding representations while enabling lightweight semantic probing. The multimodal embedding backbone is trained via contrastive learning, giving it strong representation capabilities. However, its original language modeling head suffered generative degradation. Restoring autoregressive decoding on this backbone would require unfreezing the backbone for multi-task generation, which harms the compact embedding space. Attaching a lightweight 0.6B frozen text decoder frames reconstruction as a controlled prefix rephrasing task, training only a lightweight MLP projector. This design acts as a non-intrusive probing mechanism that validates whether latent tokens encode pragmatic semantics with minimal interference to the classification manifold. The 0.6B decoder provides a cheap and lightweight solution with high engineering flexibility.

Furthermore, we tested a matched Qwen3-VL-2B multimodal architecture with full fine-tuning. Results show that when the decoder has multimodal capacity and joins full end-to-end training, latent prefixes align more closely with the multimodal annotation space. Primary performance improves to 78.66 / 80.44 (+0.74 and +0.85 points over the default lightweight setup). This confirms the capacity scalability of our latent-token framework. We will add this experiment and discussion to Appendix of the revised manuscript.

Reviewer_MgiqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The current multimodal sarcasm detection is usually a binary classification task, but the labels cannot fully describe the specific mechanism of sarcasm. Moreover, the annotation standards for different datasets are inconsistent. Therefore, the authors proposed a fine-grained sarcasm detection benchmark, and a single-tower model built on Qwen3-VL-Embedding-2B, using fine-grained annotation-guided latent-token modeling. Experiments demonstrate the effectiveness.

Critical strengths
  1. The research content is interesting.

  2. The writing of the method is good.

Critical weaknesses
  1. The author did not seem to report the results of the consistency check between manual annotation and MLLM annotation. Many annotations were used after passing the MLLM review, and it was not clear in the paper whether the MLLM annotations were reasonable or not.

  2. Although the author conducted numerous experiments using the Strict and Broad labels, these experiments mainly demonstrated the performance of the model under this new annotation standard, rather than directly proving that the design of these labels (Strict/Broad) is superior to the original binary classification labels.

  3. The author presented some cases, but did not adequately quantify how much more accurate and faithful the semantic reconstruction made the interpretation.

  4. The author demonstrated through examples how the same images and text were assigned different original labels. Could the authors further calculate the rate of inconsistent original labels in the two datasets when the image and text contents remained the same, and analyze whether these conflicts originated from differences in annotation standards? Or could the authors conduct additional experiments to more fully verify the problem?

Questions for authors

Please refer to the weakness section.

6 comments

Deer Reviewer Mgiq

We sincerely thank the reviewer for the thoughtful evaluation and constructive feedback.


Q1

Weakness 1 The author did not seem to report the results of the consistency check between manual annotation and MLLM annotation. Many annotations were used after passing the MLLM review, and it was not clear in the paper whether the MLLM annotations were reasonable or not.
Question 1 What is the consistency between human annotation and MLLM annotation?

A1

Thank you for this essential question. We conducted a double-blind human audit on 500 stratified random samples to verify our automated annotation pipeline. Our results confirm that the multi-stage MLLM annotations are highly reliable and closely align with human experts. Across all evaluated fields, inter-annotator agreement (IAA Kappa) reaches 0.8097 to 0.9551. On primary Strict and Broad sarcasm tasks, human-model accuracy reaches 97.60% and 96.80%, with Cohen's Kappa of 0.9089 and 0.9147. We report the detailed sampling scheme and agreement metrics below, and we will add the complete results to Section 2 and Appendix B.1 of the revised paper.

To test the quality of our automated pipeline, we drew a stratified random sample of N=500N = 500 records from the full 41,812 deduplicated records. We stratified the sample proportionally across all four original datasets, covering binary labels, four pragmatic types, and three evidence sources. Two independent expert annotators with backgrounds in natural language processing re-annotated these 500 records under a double-blind protocol without seeing original labels or model outputs. The agreement metrics are shown in the table below.

Evaluated FieldSample Size (NN)Inter-Annotator Agreement (IAA Kappa)Human-Model Accuracy (Acc, %)Human-Model Macro-F1 (%)Human-Model Agreement (Cohen's Kappa)
Strict Sarcasm5000.906897.6095.440.9089
Broad Sarcasm5000.955196.8095.730.9147
Text Sarcasm5000.911397.0092.740.8550
Image Sarcasm5000.809797.4081.760.6369
Evidence Source5000.880994.6079.140.8712
Pragmatic Type5000.885090.2040.010.8643

These agreement metrics yield three main conclusions.

  1. High inter-annotator agreement. The IAA Kappa across all fields ranges between 0.8097 and 0.9551, confirming high consistency and clear category boundaries (Landis and Koch, 1977).
  2. Reliable human-model consistency. On primary tasks, human-model Cohen's Kappa reaches 0.9089 for Strict and 0.9147 for Broad sarcasm, with accuracies of 97.60% and 96.80%. These numbers confirm that multi-stage automated reflection and verification achieve expert-grade reliability.
  3. Analysis of Pragmatic Type Macro-F1. Pragmatic Type achieves 90.20% accuracy and 0.8643 Cohen's Kappa, but has an unweighted Macro-F1 of 40.01% because natural pragmatic phenomena follow a long-tail distribution. In contrast, weighted F1 reaches 89.45%, showing solid agreement across primary categories.

Q2

Weakness 2 Although the author conducted numerous experiments using the Strict and Broad labels, these experiments mainly demonstrated the performance of the model under this new annotation standard, rather than directly proving that the design of these labels (Strict/Broad) is superior to the original binary classification labels.
Question 2 How does the design of Strict and Broad labels demonstrate clear superiority over original binary labels?

A2

Thank you for this insightful question. Our Strict and Broad label design provides three clear advantages over original binary labels. First, our unified schema eliminates conflicting annotation standards across datasets and removes systematic label noise during joint training and cross-dataset evaluation. Second, our hierarchical boundary separates core attitude reversal in Strict sarcasm from peripheral mockery and exaggeration in Broad sarcasm, which enables fine-grained diagnostic evaluation. Third, our schema pairs these boundaries with multi-dimensional semantic attributes to provide explicit supervision for disentangled representation learning. We detail these three advantages below.

  1. Unifying fragmented benchmark standards and eliminating contradictory labels. The primary value of our re-annotation lies in establishing a unified, consistent definition of sarcasm across multiple datasets. Original benchmarks were created independently by different teams with conflicting standards and ambiguity. As detailed in our response to Question 4, 7.37% of identical image-text pairs carry conflicting binary labels across benchmarks. As a result, models trained on one dataset often suffer decision reversals when tested on another, which distorts cross-domain evaluation. Training on mixed datasets also introduces systematic annotation noise. SarcAtlas solves this problem by providing a standardized annotation standard across all sources.
  2. Separating nuanced pragmatic acts for diagnostic evaluation. The distinction between Strict and Broad provides researchers with a fine-grained evaluation tool. We define Strict sarcasm as instances with explicit literal inversion and verbal irony. We place peripheral cases such as visual parody, hyperbole, and mild negative tone into Broad sarcasm. This separation helps researchers test whether a model understands deep semantic inversion or merely detects superficial mockery cues.
  3. Providing multi-dimensional structured supervision. SarcAtlas goes beyond scalar classification labels by annotating comprehensive structured attributes. In addition to sarcasm boundaries, we annotate shared context, modality-specific content, cross-modal relationships, causal sarcasm explanations, pragmatic types, and evidence sources. These multi-dimensional annotations provide explicit semantic constraints that help the model disentangle latent representations. They also provide the research community with a more thorough and robust evaluation benchmark.

We will expand Section 3.2 and Section 4.3 in the revised manuscript to explain this design rationale in greater detail.

Q3

Weakness 3 The author presented some cases, but did not adequately quantify how much more accurate and faithful the semantic reconstruction made the interpretation.
Question 3 Can the authors quantify the factual accuracy and faithfulness of the semantic reconstruction across the test set?

A3

Thank you for this constructive suggestion. We evaluated semantic reconstruction quality across all 5,484 samples in the full test set. Our results confirm that the decoded fields achieve high semantic similarity (BERTScore from 88.21 to 89.42) and high internal faithfulness (94.10% to 98.18%). Because each latent group uses only 3 continuous tokens as a compact bottleneck, our model prioritizes high-level pragmatic polarity and cross-modal relations (49.87% factual accuracy) while compressing peripheral low-level entity details, resulting in factual accuracy between 23.27% and 49.87%. Reconstructing discrete natural language from continuous latent embeddings is recognized as a fundamentally difficult inverse problem. [1,2] Robust mapping from continuous embeddings to text usually requires massive datasets (10510^5 to 10610^6 samples).

To measure reconstruction quality, we evaluated our model across all 5,484 samples in the full test set in the table below.

Decoded Semantic FieldBERTScore ↑Factual Accuracy (LLM-Judge) ↑Faithfulness ↑
Shared Context89.4241.03%-
Text-Unique Content88.4727.37%95.44%
Image-Unique Content88.9123.27%94.77%
Sarcasm Explanation88.2143.14%94.10%
Cross-Modal Relation88.7049.87%98.18%

These measurements reveal three main insights.

  1. Semantic fidelity. High BERTScores (88.21 to 89.42) confirm that latent tokens reliably preserve global semantic orientation.
  2. Factual accuracy insights and local semantic trade-offs. Factual accuracy ranges from 23.27% to 49.87%. Continuous latent tokens serve as a highly compressed pragmatic bottleneck with only 3 tokens per group. The model is optimized to preserve high-level semantic polarity and cross-modal opposition, as confirmed by high cross-modal relation accuracy (49.87%) and high faithfulness (94.10% to 98.18%). This limited capacity naturally discards low-level entity details, such as decoding a snowy car scene as rainy or a glass container as a bag. While these entity differences do not alter the underlying pragmatic irony, word-level LLM judge evaluations penalize them as factual errors. This demonstrates that the model captures essential sarcasm mechanisms while compressing peripheral entity details.
  3. High faithfulness. Faithfulness remains very high across all fields (94.10% to 98.18%). This proves that decoded explanations strictly match internal model predictions without contradictory statements.

References.

[1] Morris, John, et al. "Text embeddings reveal (almost) as much as text." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[2] Zhao, Haiyan, et al. "Rep2Text: Decoding Full Text from a Single LLM Token Representation." arXiv preprint arXiv:2511.06571 (2025).

We will add these full quantitative metrics and citations to Section 4.4 and Appendix C of the revised manuscript.

Reviewer_fJYHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SarcAtlas, a unified fine-grained annotation layer for multimodal sarcasm detection across MMSD, MMSD2.0, and the English and Chinese subsets of SarcNet. It extends the original binary labels with nested Strict/Broad boundaries, pragmatic mechanisms, modality-specific content, cross-modal evidence, interpretation dependencies, and explanations, produced by an LLM annotate-review-repair pipeline with expert audit of flagged cases. The authors also propose a Qwen3-VL-Embedding-2B model with five latent-token groups and a decision token, trained with multi-task prediction, semantic reconstruction, teacher-student similarity distillation, and supervised contrastive learning. Experiments cover joint training, source-specific evaluation, cross-dataset transfer, fine-grained attribute prediction, case-level interpretation, ablations, and inference efficiency.

Critical strengths
  1. SarcAtlas provides 41,812 deduplicated annotation records with multiple semantic and pragmatic fields. The nested Strict/Broad distinction is useful for studying ambiguous boundaries and neighboring phenomena such as ridicule and direct criticism.
  2. The paper shows that 34-44% of original positives fall outside both boundaries and that datasets differ systematically in pragmatic types, evidence sources, and interpretation dependencies (Figure 2, Tables B2-B5).
  3. SarcAtlas is best on all eight joint-training Strict/Broad evaluations (Table 1) and all eight cross-dataset transfer directions (Table 3). The gains on SarcNet-ZH are large, although several baselines (CLIP, MV-CLIP, HFM) use English-only text encoders and score below chance there, so part of this gain reflects the multilingual backbone.
  4. Assigning latent-token groups to shared context, modality-specific content, cross-modal relations, and pragmatic analysis is a clear design, and the three-state prototype formulation guarantees consistency between Strict and Broad predictions.
Critical weaknesses
  1. Dola Seed generates and repairs the annotations and GPT-5.5 reviews them; only the 990 records (2.4%) still flagged after re-review receive expert audit, so about 97.6% of records are never checked by a human. The "randomly selected" author review in the AI statement is not quantified, and no human-model or inter-annotator agreement is reported. Because these labels define both training and test sets, all reported scores measure agreement with the LLM annotators. No zero-shot multimodal LLM baseline is reported under the new labels either.

  2. Gains are mostly explained by the backbone rather than the framework. In Table C4, the detection-only variant (77.59/78.83 average Strict/Broad) already exceeds every baseline in Table 1 (Multi-View CLIP averages about 73.4 Strict), so most of the 4.5-point average gain comes from LoRA-tuning a 2B multilingual backbone that is 6-26 times larger than the baselines. The full framework adds only 0.33/0.76 points, removing semantic reconstruction slightly improves average Strict F1 (78.21 vs 77.92), and detection-only is better on Strict for SarcNet-EN and SarcNet-ZH. The attribute losses, teacher-student KL, and contrastive learning are not ablated individually, and no parameter-matched baseline is provided.

  3. The analysis relies on four correctly classified cases and t-SNE plots. Reconstruction quality against the annotated fields is not measured, and the decoded text comes from a frozen Qwen3-0.6B decoder conditioned on six prefix tokens, so it may reflect decoder priors. The examples contain errors (a snow-covered car scene described as rainy, a glass container called a bag), and the attribute heads are weak, with pragmatic-type Macro-F1 of 20.6-31.4 and evidence-source Macro-F1 of 41.6-54.3 (Table C3). This weakens the claim of inspectable semantic reasoning.

  4. All experiments use a single seed (42). Bootstrap intervals resample test examples but do not capture training variance. This matters because the framework's gains over detection-only are small and the SarcNet test sets are small (267 and 399 examples).

  5. Table 4 shows SarcAtlas has about 2.1B parameters, the highest latency (197.62 ms/batch), and the lowest throughput (20.24 samples/s) among compared methods; higher MFU does not translate into practical efficiency. The claimed advantage is avoiding autoregressive explanation generation, but no comparison with a generative multimodal LLM is reported.

Questions for authors
  1. How many records were in the randomly reviewed human subset and how were they selected? Can you report human-model agreement and inter-annotator agreement for Strict/Broad labels and the main fine-grained fields?
  2. Can you compare against parameter-matched baselines, or baselines built on the same LoRA-tuned backbone, to separate the effect of the backbone from that of the latent-token framework?
  3. Can you ablate the attribute losses, teacher-student KL, and supervised contrastive learning individually, and explain why removing reconstruction improves Strict F1?
  4. Can you quantitatively evaluate decoded fields (similarity to annotated fields, factual accuracy, faithfulness) on a random test subset, including incorrectly classified cases?
  5. Can you report results over multiple seeds, a zero-shot multimodal LLM baseline under the new labels, and latency against a generative MLLM that produces explanations?
11 comments

Dear Reviewer fJYH

We have carefully read your valuable review comments. We are deeply grateful that you rated our Presentation as 3 (Clear and easy to follow) and Informativeness as 3 (I gained a substantially deeper understanding of the problem or question). To address your concerns regarding Soundness (rated 2), we have conducted extensive supplementary experiments and validations.

To improve the empirical soundness of our paper, we report a 500-sample double-blind human agreement study, multi-seed variance tests, full test set semantic reconstruction metrics, and inference latency on H200 GPUs. In addition, we have added extensive baseline comparisons on the joint test set, as summarized in the table below.

Method / Baseline ModelStrict Macro-F1 (%)Broad Macro-F1 (%)
MMBT42.4838.75
DIP57.0062.79
SigLIP2 + Linear Probe60.1265.01
ViLT69.8771.53
InterCLIP w/o MEP73.7376.57
Frozen Embedding-2B + Linear Head61.5965.43
Native SFT-2B74.5074.29
Embedding-2B LoRA + Linear Head76.9879.06
SarcAtlas-2B (Ours)77.9279.59
Frozen Embedding-8B + Linear Head68.9173.53
Embedding-8B LoRA + Linear Head81.5482.91
SarcAtlas-8B (Ours)82.5284.26

These empirical comparisons show that SarcAtlas achieves the best performance (SOTA) across both 2B and 8B scales. SarcAtlas consistently outperforms traditional multimodal baselines, recent specialized architectures, and parameter-matched backbones.

We publicly commit on OpenReview that upon publication, we will open-source our dataset, code, method, and all reproduced baselines to provide a comprehensive repository for the research community.

Q1

Weakness 1 Dola Seed generates and repairs annotations, while GPT-5.5 reviews them. Only 990 flagged records (2.4%) underwent expert verification, meaning roughly 97.6% of records were never human-checked. The "randomly selected" author reviews mentioned in the AI statement were not quantified, and neither human-model agreement nor inter-annotator agreement was reported. Because these labels define both training and test sets, all reported scores measure agreement with LLM annotators. Furthermore, zero-shot multimodal LLM baselines on the new labels are missing.
Question 1 How many records were in the randomly selected human audit subset, and how were they chosen? Could you report human-model agreement and inter-annotator agreement for Strict/Broad labels and key fine-grained fields?

A1

Thank you for this constructive question. We conducted a double-blind human audit on 500 stratified random records covering all four datasets and all automated review stages (including the 97.6% automated-pass records). Our results confirm that our automated annotation pipeline achieves expert-level reliability. Across all fields, inter-annotator agreement (IAA Kappa) reaches 0.8097 to 0.9551. On primary Strict and Broad sarcasm tasks, human-model accuracy reaches 97.60% and 96.80%, with Cohen's Kappa of 0.9089 and 0.9147, and weighted F1 on pragmatic types reaches 89.45%. We report our sampling protocol and agreement metrics below, and we will add these complete details to Section 2 and Appendix B.1 of the revised paper.

To evaluate data quality, we drew a stratified random sample of N=500N = 500 records from the full 41,812 deduplicated records. We stratified the sample proportionally across all four data sources (MMSD, MMSD2.0, SarcNet-EN, and SarcNet-ZH). We also stratified across all automated review stages, covering samples passed in Round 1 (86.5%), samples passed after repair (11.1%), and samples verified by human experts (2.4%). This sampling design ensures that the 500 records fairly represent the entire dataset, including the automated pipeline records.

We then hired two independent expert annotators with backgrounds in natural language processing. The annotators re-labeled all 500 samples under a double-blind setting without seeing original labels or model predictions. The inter-annotator agreement (IAA Kappa) and human-vs-pipeline agreement metrics are shown in the table below.

Evaluated FieldSample Size (NN)Inter-Annotator Agreement (IAA Kappa)Human-Model Accuracy (Acc, %)Human-Model Macro-F1 (%)Human-Model Agreement (Cohen's Kappa)
Strict Sarcasm5000.906897.6095.440.9089
Broad Sarcasm5000.955196.8095.730.9147
Text Sarcasm5000.911397.0092.740.8550
Image Sarcasm5000.809797.4081.760.6369
Evidence Source5000.880994.6079.140.8712
Pragmatic Type5000.885090.2040.010.8643

Based on these results, we highlight three main findings.

  1. High annotation consistency. The IAA Kappa across all fields ranges between 0.8097 and 0.9551. These values fall into the highest agreement bracket (almost perfect agreement) under standard benchmarks (Landis and Koch, 1977). This confirms that our annotation guidelines are clear and reproducible.
  2. High human-model agreement. On primary sarcasm tasks, human-model Cohen's Kappa reaches 0.9089 for Strict and 0.9147 for Broad sarcasm. Human-model accuracy reaches 97.60% and 96.80%. These results verify that our multi-stage generation, reflection, and verification pipeline achieves expert-level quality.
  3. Analysis of Pragmatic Type Macro-F1. The Pragmatic Type field achieves 90.20% accuracy and 0.8643 Cohen's Kappa, but shows a lower unweighted Macro-F1 (40.01%). This gap occurs because natural sarcasm categories follow a long-tail distribution. Common sarcasm types appear frequently, while rare types have very few examples in the 500-sample test pool. Unweighted Macro-F1 gives equal weight to rare classes and is sensitive to small sample counts. In contrast, weighted F1 reaches 89.45%, showing strong agreement on primary pragmatic categories.

Q2

Weakness 2 (Part 1) Performance gains are mainly attributable to the backbone rather than the framework itself. In Table C4, the detection-only model (average Strict/Broad F1 of 77.59/78.83) already outperforms all baselines in Table 1 (Multi-View CLIP average Strict F1 is ~73.4). Hence, the ~4.5 point gain primarily comes from LoRA tuning of the 2B multilingual backbone (6x to 26x larger than baselines). The full framework only adds 0.33/0.76 points. Parameter-matched baselines are missing.
Question 2 Can you compare against parameter-matched baselines or baselines built on the same LoRA-tuned backbone to decouple the impact of the backbone from that of the latent-token framework?

A2

Thank you for this constructive question. We trained parameter-matched baselines on the same 2B and 8B backbones under identical LoRA budgets and learning rates (LR=3×10−4LR = 3 \times 10^{-4}) to isolate our framework contribution. Our results confirm that SarcAtlas consistently outperforms parameter-matched baselines across model scales. On the 2B backbone, SarcAtlas-2B outperforms generative instruction tuning (Native SFT-2B) by 3.42 Strict F1 and 5.30 Broad F1, and beats the pure LoRA baseline across joint, in-domain, and out-of-domain evaluations. When scaled to 8B, SarcAtlas-8B widens its lead over the 8B pure LoRA baseline (+0.98 Strict F1, +1.35 Broad F1, and +1.32 out-of-domain Strict AUROC), demonstrating that our latent-token framework provides orthogonal algorithmic gains beyond backbone scaling.

We summarize model performance on Joint Training (Joint Macro-F1), In-Domain Average Macro-F1, and Out-of-Domain Average AUROC in the table below.

Model and Training ParadigmParametersJoint Macro-F1 (Strict / Broad)In-Domain Avg Macro-F1 (Strict / Broad)Out-of-Domain Avg AUROC (Strict / Broad)
Frozen Embedding-2B + Linear Head2B61.59 / 65.4366.15 / 69.3164.37 / 68.82
Native SFT-2B (Generative SFT)2B74.50 / 74.2975.76 / 75.7880.72 / 82.49
Embedding-2B LoRA + Linear Head (Pure LoRA Baseline)2B76.98 / 79.0677.65 / 78.6381.49 / 82.06
SarcAtlas-2B (Ours)2B77.92 / 79.5978.45 / 78.7381.80 / 82.64
Frozen Embedding-8B + Linear Head8B68.91 / 73.5369.50 / 73.6175.51 / 79.42
Embedding-8B LoRA + Linear Head (8B Pure LoRA Baseline)8B81.54 / 82.9181.98 / 82.6886.35 / 88.02
SarcAtlas-8B (Ours)8B82.52 / 84.2682.52 / 84.2687.67 / 88.55

(Note. In-domain averages reflect the arithmetic mean across MMSD, MMSD2.0, and SarcNet EN+ZH test sets. Out-of-domain averages reflect the arithmetic mean across all cross-domain transfer directions.)

Our experimental results lead to three key observations.

  1. Backbone capacity and hardware utilization. Smaller models with fewer than one billion parameters suffer from memory-bandwidth bottlenecks on modern GPUs such as NVIDIA H100 and H200. These small models achieve low Model Flops Utilization (MFU). They also lack broad world knowledge, resulting in weak out-of-domain transfer (AUROC between 60% and 70%). Modern 2B and 8B backbones provide the necessary capacity for cross-domain transfer while saturating GPU compute.
  2. Advantages over generative baselines (Native SFT-2B). On the identical 2B backbone, generative instruction tuning achieves 74.50 / 74.29 Strict and Broad F1. SarcAtlas-2B reaches 77.92 / 79.59, outperforming Native SFT-2B by 3.42 and 5.30 points. This comparison shows that latent-token routing and prototype losses form clearer decision boundaries than autoregressive token generation.
  3. Consistent algorithmic framework gains. SarcAtlas-2B consistently outperforms the pure LoRA baseline under identical parameter budgets (+0.94 Joint Strict, +0.53 Joint Broad, and +0.80 In-domain Strict). When scaled to 8B, SarcAtlas-8B widens this performance lead, exceeding the 8B pure LoRA baseline by +0.98 Strict F1 and +1.35 Broad F1. SarcAtlas-8B also improves out-of-domain AUROC to 87.67 / 88.55 (+1.32 and +0.53). These gains confirm that our latent-token architecture provides orthogonal improvements across model scales.

We will add these controlled baselines to Table 1 in Section 4 of the revised paper.

Comments on the paper

2 comments

Dear Reviewers fJYH, Mgiq, and tDTb,

We sincerely thank you for your careful review and valuable suggestions. Your feedback reflects deep research experience and has provided clear guidance for our additional experiments and manuscript revisions. We have made the following changes, mainly in the appendices. Revised text is marked in blue.

  • Human evaluation (fJYH Q1, Mgiq Q1, and tDTb Q1). We added an independent human evaluation on 500 stratified random records. We report inter-annotator agreement and agreement between human adjudication and the annotation pipeline.
  • Label boundaries (Mgiq Q2 and Q4, and tDTb Q1). We clarified how Original labels are preserved and the roles of Strict and Broad.
  • Backbone comparisons and training stability (fJYH Q2 and Q5). We added baselines using the same 2B and 8B backbones. We also report results across three random seeds, together with sample standard deviations.
  • Training objectives and model design (fJYH Q3 and tDTb Q2). We added ablations of individual loss components and an experiment with a larger reconstruction decoder. We clarified the roles of the representation, decision, and reconstruction modules.
  • Reconstruction quality (fJYH Q4 and Mgiq Q3). We report semantic similarity, factual accuracy, and prediction consistency across all 5,484 test records.
  • Comparisons with generative models (fJYH Q5 and tDTb Q2). We added zero-shot multimodal baselines and compared direct classification with explanation-first prompting. We also report latency and throughput when models generate explanations of different lengths.
  • Architecture diagram (tDTb Q3). We added a revised architecture diagram to the appendix. The diagram shows component connections and dataflow and identifies trainable and frozen modules.

We especially thank Reviewer tDTb for the positive assessment of our contributions and the careful, generous, and constructive feedback. The ICLR community continues to grow through reviewers who uphold academic standards and devote time to supporting their peers. We value the suggestions of all three reviewers. We are also actively applying for more experimental resources to conduct further validation and improve the manuscript to meet your expectations.

As the rebuttal period is nearing its end, we sincerely hope you will reconsider your evaluation and consider raising your score if these revisions have addressed your main concerns. Your recognition and support mean a great deal to us and to this work. Wishing you all the best and every success in your own ongoing research.

Best regards,
The Authors

Dear AC and Reviewers,

We sincerely thank the AC and all three reviewers for your hard work, patience, and insightful guidance throughout the review process. Below is a brief summary of our core contributions and the rebuttal outcome.

Our paper makes three main contributions.

  1. A unified, fine-grained annotation layer (SarcAtlas). By unifying divergent decision boundaries across four benchmarks into nested Strict and Broad criteria with rich pragmatic and modal annotations, SarcAtlas resolves cross-dataset label conflicts and enables reliable data scaling and cross-dataset evaluation.
  2. Efficient and inspectable latent-token modeling. Built on a single-tower Qwen3-VL-Embedding backbone with specialized latent-token groups and frozen prototypes, our model combines the generalization and interpretability of generative models with the inference efficiency and hardware utilization of classification models.
  3. Strong empirical validation. Extensive controlled experiments confirm consistent gains over parameter-matched baselines, generative fine-tuning, and frontier zero-shot MLLMs across model scales and random seeds.

We are deeply encouraged by the positive interaction with all reviewers. Reviewer tDTb, a domain expert, consistently gave a high score of 4 with high confidence (4). After we conducted thorough additional experiments and updated our manuscript (highlighted in blue in the revised PDF), both Reviewer Mgiq and Reviewer fJYH had their concerns largely addressed and raised their scores from 2 to 3, bringing our average score to 3.33 (4, 3, 3). We are continuing to polish the manuscript to meet every expectation of the reviewers.

Once again, thank you all for your dedicated service and generous recognition. Although multimodal sarcasm is not as popular as topics like coding, your careful engagement shows the community's genuine respect for every piece of peer work. We wish you all the best in your research and future endeavors.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.