acceptodds
Under review as a conference paper at ICLR 2027

IRAC: Retrieval-Augmented Multi-Agent Collaboration for Multimodal Sarcasm Detection

Abstract

Multimodal Sarcasm Detection (MSD) aims to identify ironic intent in image–text pairs, where sarcasm typically arises from inconsistencies or semantic incongruity between linguistic and visual cues. However, most MSD research still relies on coarse binary labels, and even recent efforts that enrich annotations seldom identify which modality drives the irony—that is, its underlying source. This limitation leads models to fit shallow statistical correlations rather than understanding deep cross-modal interactions, while traditional supervised methods suffer from restricted generalization in open-world scenarios. In this paper, we present MIRS, a large-scale benchmark for Multimodal Irony Reasoning and Source-attribution, containing 11,000 samples with fine-grained four-layer hierarchical annotations. To tackle the demands of zero-shot explainable analysis—requiring both precise detection and human-aligned diagnostics—we propose IRAC (Irony-oriented Retrieval-Augmented multi-agent Collaboration). IRAC employs a specialized swarm of agents (Textual, Visual, and Cross-modal) and an orchestrator optimized via Group Relative Policy Optimization (GRPO) with irony-specific rewards. Experimental results demonstrate that IRAC achieves state-of-the-art performance (86.92% Acc) and significantly reduces hallucination rate, providing a new paradigm for explainable multimodal irony analysis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reinforcement Learning, Multimodal Sarcasm Detection, Multi-Agent Collaboration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dC3aOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MIRS, a benchmark containing 11,000 image–text pairs with hierarchical annotations for multimodal sarcasm detection, including sarcasm labels, modality attribution, and explanations. It also proposes IRAC, a retrieval-augmented framework comprising textual, visual, and cross-modal reasoning agents, coordinated by an orchestrator trained with GRPO. The proposed reward combines prediction accuracy, confidence, attribution, and consistency. Experiments on MIRS evaluate both classification performance and explanation quality.

Critical strengths

The main strength is the potentially valuable dataset contribution. MIRS extends conventional binary sarcasm detection with modality attribution and source-specific rationales, which could facilitate more detailed analysis of cross-modal incongruity and model failure modes. Its collection from Weibo, Reddit, and existing datasets potentially provides broader linguistic and cultural coverage, while the reported filtering and human verification procedures are useful efforts toward ensuring data quality.

Critical weaknesses
  1. The performance advantage is limited: IRAC achieves 86.92% accuracy compared with 86.08% for GPT-4o with retrieval augmentation, while IRAC without GRPO achieves only 85.79%. Without explicit variability estimates or significance tests, these results do not convincingly establish the benefit of the multi-agent design. 2. The comparisons do not adequately control for backbone strength or inference budget. The specialized agents use GPT-5.4, but no standalone GPT-5.4 baseline with the same retrieval context is included. Multiple model calls and additional training further complicate attribution of the gains. 3. The methodological novelty appears limited. The framework largely combines widely used techniques, including retrieval augmentation, prompt-based agent specialization, and GRPO, without a sufficiently distinctive motivation or task-specific design. The paper also does not clearly establish what generalizable methodological insights this combination offers to the broader multimodal reasoning or multi-agent learning community.
Questions for authors
  1. Have you considered the relationship between sarcasm, irony, and humor when constructing MIRS? How are these concepts distinguished, and are humorous but non-sarcastic examples explicitly included to prevent visual absurdity or playful incongruity from being mistaken for sarcasm? 2. Could you clarify the annotation procedure? Although Appendix A.6.4 describes LLM-generated drafts followed by human verification, the main text provides limited detail. Which model generated the drafts, how many annotators reviewed each sample, were judgments made independently, how were disagreements resolved, and what was the inter-annotator agreement for sarcasm labels and modality attribution? 3. How does IRAC compare with standalone GPT-5.4 using the same knowledge base, and with a single-model multi-call approach under a comparable inference budget?
Reviewer_w14wOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces MIRS, an 11,000-example image-text benchmark for sarcasm detection with hierarchical annotations intended to include modality attribution and rationales. It proposes IRAC, which retrieves contextual examples, delegates textual, visual, and cross-modal analysis to specialized agents, and trains an orchestrator with a GRPO-based reward. The method is evaluated on MIRS for classification and rationale-quality metrics.

Critical strengths
  • The paper clearly decomposes multimodal sarcasm analysis into textual, visual, and cross-modal evidence, exploring the distinct roles of unimodal and cross-modal interaction cues in sarcasm detection.
  • The paper introduces MIRS, a new benchmark with 11,000 image-text samples and fine-grained annotations, and publicly releases both the dataset and code to support future research and reproducibility.
Critical weaknesses
  • The methodological novelty is limited, as the proposed framework mainly combines existing techniques, including retrieval augmentation, multi-agent collaboration, weighted aggregation, and GRPO, without introducing substantial new methodological innovations.
  • The experiments do not sufficiently isolate the contributions of retrieval, agent decomposition, and GRASP, making it difficult to determine which components drive the performance gains. In addition, the paper lacks direct comparisons with existing retrieval-augmented multimodal sarcasm detection methods.
  • The performance gains over strong existing models are relatively modest, despite the additional complexity of the proposed framework. This raises concerns about whether the limited improvements justify the additional task-specific training and multi-agent inference overhead.
Questions for authors

See Weaknesses.

Reviewer_2c6cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MIRS, an 11K image-text sarcasm dataset with fine-grained modality attribution and explanation annotations. It further proposes IRAC, a retrieval-augmented multi-agent framework with specialized textual, visual, and cross-modal agents. A GRPO-trained orchestrator learns to aggregate their predictions using irony-specific rewards. Experiments on MIRS report competitive sarcasm detection performance and improvements in explanation quality and attribution consistency.

Critical strengths
  1. The paper explores an interesting direction by identifying the sources of multimodal sarcasm rather than relying solely on binary classification. MIRS provides potentially useful fine-grained annotations.

  2. The framework combines external knowledge with modality-specific reasoning. The experiments include multiple MLLMs, retrieval comparisons, and evaluations of rationale quality.

Critical weaknesses
  1. Although MIRS contains human-verified attribution labels, the main Attribution Consistency metric measures whether larger weights are assigned to agents making correct predictions. This is closely related to the attribution reward used during training and does not directly validate agreement with human-annotated sarcasm sources.

  2. The three specialized agents use GPT-5.4, while the orchestrator uses Qwen3-VL-4B. Compared with single-model baselines, IRAC benefits from multiple powerful model calls and external knowledge, making it difficult to isolate the contribution of multi-agent orchestration.

  3. Retrieval augmentation, specialized agents, dynamic weighting, and GRPO are established techniques. The improvement from IRAC Base to IRAC GRPO is only 1.13 percentage points in accuracy, and the individual contributions of the GRASP reward components are not sufficiently evaluated.

  4. Section 3.5 states that the orchestrator can override the weighted prediction, while the appendix instructs it to determine the final label using a fixed 0.5 threshold. This ambiguity affects the interpretation of the consistency reward.

  5. The experiments rely mainly on the newly constructed MIRS benchmark, without independent cross-dataset validation. The rationale evaluation also uses GPT-5.4 as a judge, while the same model powers the specialized agents, raising concerns about evaluator dependence.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.