acceptodds
Withdrawn from ICLR 2027

Meaning Beyond Language: Learning Complementary Semantics for Multimodal Metaphor Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Metaphor is widely used to express meanings beyond literal content. Multimodal metaphor detection aims to identify metaphorical meaning by jointly understanding text and images. Although existing methods have made notable progress, they mainly rely on a single language and overlook the complementary semantic information expressed in other languages. To address this limitation, we propose \ourmethod, a framework that uses an auxiliary language to improve multimodal metaphor detection. First, we introduce a cross lingual semantic disentanglement module to separate shared semantics from language specific information. Contrastive learning, reconstruction, and orthogonality constraints are used to preserve useful information and reduce the effect of language differences. Second, since different vision language model layers capture different semantic cues, we fuse the representations of both languages at each layer through self attention. We then assign different weights to these layers based on the input sample, allowing the model to focus on the semantic cues most relevant to metaphor detection. Finally, to address the absence of translations during inference, we learn a mapping that predicts the semantic representation of the translation from that of the original text. Experiments on multiple benchmarks with different vision language models demonstrate the effectiveness of our framework.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Metaphor Detection, Cross-Lingual Learning, Language-Agnostic Representations, Representation Disentanglement, Activation Steering, Vision-Language Models, Cross-Cultural Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6V6DOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LAMCC, a framework that leverages semantic information from an auxiliary language to enhance performance in multimodal metaphor detection. The method extracts shared semantics and language-specific information through cross-lingual semantic decoupling, improves metaphor prediction via self-attention-based fusion and adaptive layer selection, and simulates auxiliary language representations during inference using linear mapping. Experiments conducted on two datasets across four vision-language models demonstrate strong performance.

Critical strengths
  1. The paper explores an interesting direction for multimodal metaphor detection by incorporating information from an auxiliary language. The motivation is reasonable and relevant to the task.
  2. The proposed method combines semantic disentanglement, attention-based fusion, and adaptive layer selection. The semantic simulation module is also useful, as it avoids requiring translations at inference time.
  3. The experiments cover two datasets and four VLM backbones, with consistent improvements over SFT. The ablation results also help demonstrate the contributions of the main components.
Critical weaknesses
  1. The paper does not clearly describe how the training, validation, and test sets are split, or how the Chinese and English samples are organized. More details are needed to ensure reproducibility.
  2. Although LAMCC achieves the best results, the improvements over some strong baselines are relatively small. For example, the gains over Semantic Disentangled CC-Tuning on MultiMM are only 0.68% and 0.24%. The paper also lacks results from multiple random seeds, making it difficult to assess the significance of these improvements.
  3. The paper does not provide enough evidence that the improvements come from complementary cross-lingual semantics. Additional comparisons with translation-augmented fine-tuning would help support this claim.
  4. The formulation of the Decision Maker in Eq. (10) is somewhat unclear. Since the representations from different layers are averaged before computing the layer weights, it is unclear how the model identifies the importance of individual layers.
  5. The semantic simulation module is not sufficiently evaluated. A comparison between real auxiliary representations, predicted representations, and directly reused main-language representations would help clarify its actual contribution. 6.The paper lacks sufficient visualization and analysis of its key components. Although Figure 3 provides some case studies and layer-wise fusion weights, it does not clearly demonstrate the effectiveness of semantic disentanglement, adaptive layer selection, or semantic simulation. More detailed representation visualizations and layer-wise analyses would help support the proposed mechanisms. In addition, most of the case studies focus on Chinese examples, while examples in English are limited.
Questions for authors
  1. Could the authors provide more details about the training, validation, and test splits? How are the Chinese and English samples organized, and how is data leakage between the splits avoided?
  2. The improvements over some strong baselines are relatively small. Could the authors report results across multiple random seeds to demonstrate the stability of these improvements?
  3. How can the authors demonstrate that the performance gains come from complementary cross-lingual semantics rather than additional training data or model complexity?
  4. In Eq. (10), the representations from different layers are averaged before computing the layer weights. How does the Decision Maker distinguish the importance of individual layers?
  5. Could the authors compare the performance using real auxiliary-language representations, simulated representations, and directly reused main-language representations to better evaluate the semantic simulation module?
Reviewer_f4SdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LAMCC, a framework for multimodal metaphor detection that leverages an auxiliary language to improve semantic understanding. Experiments on MultiMM and MET-MEME show improvements over supervised fine-tuning across several vision-language models.

Critical strengths
  • The paper explores an interesting direction by investigating whether different languages can provide complementary information for multimodal metaphor detection.

  • The experiments cover two benchmarks and four VLM backbones, with consistent improvements over supervised fine-tuning.

Critical weaknesses
  • The overall framework is closely related to[1]. Both methods establish connections between representations obtained from different languages, use a Decision Maker to select useful information, inject the selected representations into the main model, and learn a transformation to approximate auxiliary representations during inference. In Table 1, LAMCC improves over Semantic Disentangled CC-Tuning by only 0.68 and 0.24 percentage points on the Chinese and English splits, respectively.

  • The paper does not clearly demonstrate that the improvements come from complementary cross-lingual semantics, since the two language representations are explicitly aligned and the auxiliary representation is reconstructed from the original one at inference time.

  • There is a concern regarding the reported evaluation metrics. Table 1 labels F1-M as macro-F1, but several baseline results report macro-F1 values substantially higher than accuracy.

Ref: [1] CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning. ACL 2025

Questions for authors
  • What is the main methodological contribution of LAMCC beyond CC-Tuning, and which experimental results establish the necessity of the proposed changes?

  • How does the Decision Maker in Eq. (10) assess the relevance of individual layers when its input is aggregated across all layers?

Reviewer_nbsNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LAMCC uses image-conditioned semantic decomposition, bilingual attention fusion, and layer weighting for multimodal metaphor detection. It fits a ridge map using representations from a calibration set of 1,000 training samples to approximate translated features at inference, so the model needs only the original input. Experiments cover two datasets, two languages, and four Qwen backbones. Within the acknowledged CC-Tuning framework, the main changes are the decomposition and fusion modules, rather than layer selection or translation simulation itself (§2; Appendix A).

Critical strengths
  • Table 3 reports accuracy gains over SFT in all 16 backbone–language–dataset settings. Table 4 also includes module ablations for disentanglement and layer selection. The four backbones cover two Qwen version series; they are not independent random repetitions.
  • Training with bilingual data while using original-input inference is a clear deployment setup. Table 1 compares CC-Tuning, its multimodal auxiliary variant, and a semantic-disentangled variant constructed by the authors. The last control tests what the full design adds beyond decomposition.
Critical weaknesses
  • Section 3.2 calls Table 1 macro F1, but LLaVA Chinese has F1 69.84 versus accuracy 56.81. For standard binary metrics computed from the same predictions, samples, and sample weights, macro F1 ≤ accuracy. The original MultiMM Table 5 contains identical LLaVA/SEMD entries, so this is partly inherited reporting, not a demonstrated new miscalculation. Positive-class F1 or different denominators could explain it. Section 3.1 and Appendix B also leave the prompts, split IDs, baseline provenance, and source/use of sentiment unclear. These missing details limit cross-method comparisons; unclear sentiment access alone is not evidence of leakage.
  • The gain over the authors' semantic-disentangled control is 0.68/0.24 accuracy points in Chinese/English (Table 1), with one reported seed (Appendix B). Attention also lowers English MultiMM accuracy from 90.90 to 90.41 in Table 4. A real-translation versus ridge-simulation comparison is missing, leaving the inference approximation unquantified. Since the auxiliary feature is a deterministic linear map of the original representation, the results do not separate translation-specific complementary information from regularization. The orthogonality loss in Eq. 5 does not guarantee statistical independence.
Questions for authors
  • Could the authors reconcile Table 1's F1 definitions and sources, and specify each method's prompts, sentiment access, split IDs, and reused or rerun results?
  • Do paired predictions or repeated runs support the 0.68/0.24-point gain? What is the gap between real translated features and ridge simulation, and which mechanism claims remain supported?
  • Could the authors further relate binary multimodal metaphor detection to the implication-understanding tasks in II-Bench/CII-Bench and the visual-metaphor tasks in MetaCLUE/I Spy a Metaphor? Please also discuss LAMCC's bilingual latent fusion and translation-feature simulation alongside MetaphorStar's TFQ-GRPO training and Let Androids Dream's perception–search–reasoning pipeline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.