- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces SarcAtlas, a fine-grained multimodal sarcasm resource and a latent-token framework that uses structured annotations to jointly support sarcasm detection, attribute prediction, and semantic reconstruction.
Critical strengths
- The authors conduct additional annotations that well align with the method’s motivation.
- The framework combines efficient classification through a single multimodal forward pass with optional text reconstruction, allowing intermediate semantic representations to be inspected without requiring autoregressive generation for detection.
- The paper is well written and presents solid work.
Critical weaknesses
-
The annotations are produced through an almost entirely automated pipeline, with only the 2.4% of samples that failed to pass automated review undergoing expert verification. Manual checks for cases that passed ROUND 1 and 2 seemed to be missing to ensure the automated annotation pipeline is reliable.
-
The validation of the semantic information captured by the latent-token groups is limited. While semantic reconstruction appears reasonable, the rationale of design is not sufficiently explained, particularly the use of different models (Qwen3-VL-Embedding-2B & Qwen3-0.6B) for representation extraction and text decoding, and the choice to use embedding-based predictions rather than direct decoding despite adopting a generative backbone.
-
A minor point regarding illustration: Figure 3 provides a detailed breakdown of the training components, but a clearer overview of the complete model architecture would make the method easier to follow. eg: The Figure highlights LoRA, but does not clearly identifying the other trainable components, like prefill tokens and the projector, which can be a little bit confusing for readers at first. This is a minor presentation issue that affects readability rather than a fundamental weakness of the approach.
Questions for authors
- Figure 1 presents four categories. Does category D, “Noise,” refer to incorrect annotations in the original dataset? However, the annotation schema in Section 2 states that the Original label is retained. Are the labels verified and corrected during annotation? What is the specific procedure?
- Why is Qwen3-VL-Embedding-2B used for representation extraction, while Qwen3-0.6B is used for reconstruction? I think the reason should be more clearly specified in the Method section.
5 comments
Q1
Weakness 1 The annotations are produced through an almost entirely automated pipeline, with only the 2.4% of samples that failed to pass automated review undergoing expert verification. Manual checks for cases that passed ROUND 1 and 2 seemed to be missing to ensure the automated annotation pipeline is reliable.
Question 1 Figure 1 shows four categories. Does Category D "Noise" refer to mislabeling in the original datasets? However, the annotation scheme in Section 2 states that original labels are preserved. Are labels verified and corrected during annotation, and what is the specific process?
A1
Thank you for your thorough review and constructive questions. Below, we report our human verification experiments, the definition of Category D in Figure 1, and our handling of original labels.
To verify samples that passed automated review, we conducted a quantitative audit and will add it to Appendix B.1. We drew a stratified random sample of records from the full 41,812 deduplicated records across all four sources. Two independent experts with backgrounds in natural language processing re-annotated these 500 samples under a double-blind protocol without seeing original labels or model outputs. The agreement metrics are summarized in the table below.
| Evaluated Field | Sample Size () | Inter-Annotator Agreement (IAA Kappa) | Human-Model Accuracy (Acc, %) | Human-Model Macro-F1 (%) | Human-Model Agreement (Cohen's Kappa) |
|---|---|---|---|---|---|
| Strict Sarcasm | 500 | 0.9068 | 97.60 | 95.44 | 0.9089 |
| Broad Sarcasm | 500 | 0.9551 | 96.80 | 95.73 | 0.9147 |
| Text Sarcasm | 500 | 0.9113 | 97.00 | 92.74 | 0.8550 |
| Image Sarcasm | 500 | 0.8097 | 97.40 | 81.76 | 0.6369 |
| Evidence Source | 500 | 0.8809 | 94.60 | 79.14 | 0.8712 |
| Pragmatic Type | 500 | 0.8850 | 90.20 | 40.01 | 0.8643 |
Across all fields, IAA Kappa ranges between 0.8097 and 0.9551, indicating almost perfect agreement. On primary tasks, human-model Cohen's Kappa reaches 0.9089 for Strict and 0.9147 for Broad sarcasm with 97.60% and 96.80% accuracy. These numbers confirm that our multi-stage reflection and verification pipeline produces reliable annotations.
Regarding Category D "Noise" in Figure 1, Category D represents data noise and pseudo-positives from legacy datasets, with a prominent subcategory being "Mention, not use". Original datasets were collected using keyword and hashtag queries (such as searching #sarcasm). This collection method collected posts that contain the word "sarcasm" or meme templates without expressing actual sarcasm. Examples include personality memes listing personal traits (Table B6 Case d) or posts discussing the concept of sarcasm without using it (Figure 1 Case D). These instances contain no irony, attitude reversal, or mockery. Legacy datasets mislabeled them as positive, creating keyword shortcuts for models. In SarcAtlas, we label the pragmatic type of this subcategory as Sarcasm mention and assign both Strict and Broad ground truths to non-sarcasm (Neither), removing these false positives.
Regarding the statement in Section 2 about preserving original labels, this refers to preserving metadata for historical reference. We retain original binary label fields in dataset records strictly for historical benchmarking and auditing. Our annotation workflow operates as follows. Tool-augmented multimodal models generate initial structured annotations. Independent multimodal models conduct logical verification. Non-conforming samples undergo targeted repair, and difficult edge cases (2.4%) are sent to human experts. Together with our 500-sample double-blind audit, this forms a closed quality-control process. We will clarify this wording in Section 2 and Figure 1 of the revised manuscript.
Q2
Weakness 2 The validation of the semantic information captured by the latent-token groups is limited. While semantic reconstruction appears reasonable, the rationale of design is not sufficiently explained, particularly the use of different models (Qwen3-VL-Embedding-2B & Qwen3-0.6B) for representation extraction and text decoding, and the choice to use embedding-based predictions rather than direct decoding despite adopting a generative backbone.
Question 2 Why is Qwen3-VL-Embedding-2B used for representation extraction, while Qwen3-0.6B is used for reconstruction?
A2
Thank you for this question. Our design choices are guided by two core factors, decision reliability and inference efficiency.
First, evaluating decision reliability shows that requiring models to generate explanations before predicting labels introduces narrative hallucinations and rationalization bias. As shown in the table below, Qwen3-VL-32B achieves 67.26 Strict F1 in direct classification, but its Strict F1 drops to 56.19 in explain-then-classify mode (-20.83% accuracy). Similarly, generative Qwen3-VL-2B reaches only 54.87 / 54.29 Strict F1, and generative fine-tuning (Native SFT-2B with 74.50 / 74.29) lags behind SarcAtlas. Latent-token single-forward classification isolates the decision manifold from free-text generation errors.
| Model / Architecture | Inference Mode | Strict Macro-F1 (%) | Broad Macro-F1 (%) | Strict Accuracy (%) | Broad Accuracy (%) |
|---|---|---|---|---|---|
| Qwen3-VL-2B-Instruct | Direct classification | 54.87 | 49.67 | 65.90 | 50.26 |
| Qwen3-VL-2B-Instruct | Explain then classify | 54.29 | 58.66 | 55.76 | 59.12 |
| Qwen3-VL-4B-Instruct | Direct classification | 51.38 | 54.73 | 73.22 | 66.92 |
| Qwen3-VL-4B-Instruct | Explain then classify | 57.14 | 62.69 | 58.36 | 63.24 |
| Qwen3-VL-8B-Instruct | Direct classification | 61.40 | 62.06 | 73.09 | 69.58 |
| Qwen3-VL-8B-Instruct | Explain then classify | 61.76 | 66.71 | 63.03 | 67.06 |
| Qwen3-VL-32B-Instruct | Direct classification | 67.26 | 74.72 | 77.25 | 77.56 |
| Qwen3-VL-32B-Instruct | Explain then classify | 56.19 | 63.64 | 56.42 | 63.95 |
| Native SFT-2B | Direct classification | 74.50 | 74.29 | 78.85 | 76.53 |
| SarcAtlas-2B (Ours) | Single forward pass | 77.92 | 79.59 | 83.38 | 80.63 |
Second, evaluating engineering efficiency on NVIDIA H200 GPUs demonstrates that single-forward classification provides major throughput gains. Modern GPUs achieve high compute utilization during dense prefill operations (SarcAtlas achieves 46.97% MFU). In contrast, autoregressive generation is limited by memory bandwidth. As shown in the table below, SarcAtlas processes 35.39 samples per second with a latency of 113.02 ms per batch. When generative models produce explanations, throughput drops to 1.34 to 4.81 samples per second and latency rises to 830 to 2979 ms. SarcAtlas delivers a 7.3x to 26.4x throughput advantage.
| Method / Inference Mode | Output Content and Format | Throughput (samples/s) ↑ | Latency (ms/batch) ↓ |
|---|---|---|---|
| SarcAtlas-2B (Ours) | Deterministic labels + all fine-grained attributes | 35.39 | 113.02 |
| Qwen3-VL-2B (Generative) | Label only | 30.54 | 130.99 |
| Qwen3-VL-2B (Generative) | 32-token short explanation + label | 4.81 | 830.82 |
| Qwen3-VL-2B (Generative) | 64-token medium explanation + label | 2.60 | 1536.32 |
| Qwen3-VL-2B (Generative) | 128-token detailed explanation + label | 1.34 | 2979.81 |
Regarding model selection, decoupling Qwen3-VL-Embedding-2B from a frozen Qwen3-0.6B decoder protects embedding representations while enabling lightweight semantic probing. The multimodal embedding backbone is trained via contrastive learning, giving it strong representation capabilities. However, its original language modeling head suffered generative degradation. Restoring autoregressive decoding on this backbone would require unfreezing the backbone for multi-task generation, which harms the compact embedding space. Attaching a lightweight 0.6B frozen text decoder frames reconstruction as a controlled prefix rephrasing task, training only a lightweight MLP projector. This design acts as a non-intrusive probing mechanism that validates whether latent tokens encode pragmatic semantics with minimal interference to the classification manifold. The 0.6B decoder provides a cheap and lightweight solution with high engineering flexibility.
Furthermore, we tested a matched Qwen3-VL-2B multimodal architecture with full fine-tuning. Results show that when the decoder has multimodal capacity and joins full end-to-end training, latent prefixes align more closely with the multimodal annotation space. Primary performance improves to 78.66 / 80.44 (+0.74 and +0.85 points over the default lightweight setup). This confirms the capacity scalability of our latent-token framework. We will add this experiment and discussion to Appendix of the revised manuscript.
Dear Reviewer tDTb
We express our heartfelt gratitude to the reviewer for the highly positive evaluation, generous encouragement, and constructive guidance.
We sincerely appreciate your recognition of our fine-grained annotation effort, our single-forward efficiency, and our decoupled text reconstruction design. Your positive assessment that our paper presents well-written and solid work provides tremendous motivation for our team.
Your insightful questions regarding automated pipeline audits, legacy dataset noise, and illustration clarity have provided valuable direction for refining our paper. We have conducted extensive supplementary experiments and prepared detailed clarifications to address every point thoroughly. We present our systematic responses and empirical data below. Thank you again for your time, support, and dedication.