acceptodds
Withdrawn from ICLR 2027

Retrieval-Augmented Label-Attentive Prompt Learning for Multimodal Classification with Missing Modalities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Missing modalities make multimodal classifiers brittle because the available evidence is partial, while common remedies either synthesize the missing modality or rely on static missing-aware prompts. We argue that missing-modality classification should instead be treated as retrieval-augmented evidence selection: a memory bank can provide useful neighbors, but the retrieved labels and embeddings are noisy and should not be trusted uniformly. We introduce Retrieval-Augmented label-attentive prompt Learning for Multimodal Classification with Missing Modalities (RALAP), a frozen-LLM framework that converts retrieved multimodal neighbors into dynamic soft prompts. RALAP first encodes the target self-embeddings and target–neighbor difference embeddings through a sparse autoencoder bottleneck to generate differential soft prompts. It then computes a label-attentive prompt by weighting retrieved instances with a similarity kernel and aggregating their labels into a target conditioned label prior. The resulting self, difference, and label prompts are injected into a frozen Qwen2-VL model, which jointly generates a class label and a short explanation. We further show theoretically that, with sufficient retrieval coverage and stronger aggregated evidence for ground-truth labels than distractors, ground-truth labels receive higher attention scores. Empirically, the results support the need for target conditioned reweighting and are consistent with the theoretical behavior of label attention. RALAP-VLM achieves the best reported performance in 10 of 12 evaluation settings across MM-IMDb, HateMemes, and Food101, with a maximum observed improvement of 21.84 percentage points on MM-IMDb. These results support the effectiveness of combining retrieval-enhanced sparse differential prompts with target conditioned label-attentive prompts under modality missingness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Missing-Modality Multimodal Learning, Retrieval-Augmented Prompting, Dynamic Soft Prompts, Label Attention, Multimodal Classification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h2Q3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Retrieval-Augmented label-attentive prompt Learning framework for Multimodal Classification with Missing Modalities (RALAP), a frozen-LLM framework that converts retrieved multimodal neighbors into dynamic soft prompts.

Critical strengths

The addressed problem, modality-missing learning, is very important. And the multimodal transformer is a good research body.

Critical weaknesses

The primary motivation for studying missing modalities in multimodal Transformers is to develop lightweight, non-invasive adaptation methods that maintain model robustness under incomplete modality conditions. However, this paper directly introduces Qwen2-VL-7B as its backbone, which fundamentally contradicts the lightweight adaptation philosophy underlying this research direction and substantially undermines the practical significance of the work. The authors appear to have insufficiently understood the core motivation of this research problem. Notably, citing PROMISE (AAAI 2026) as a precedent would not resolve this fundamental concern, as PROMISE suffers from the same issue. The proposed approach therefore deviates from the central objective of this research direction, and its necessity and practical value remain unconvincing.

Questions for authors

See weaknesses.

Reviewer_pJtLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RALAP for classification when one modality is missing. It retrieves complete multimodal neighbors, encodes the target and target-neighbor differences as soft prompts, and uses retrieved labels to form a target-conditioned label prompt for a frozen Qwen2-VL model. The model predicts a label and an explanation. Experiments on MM-IMDb, HateMemes, and Food101 report strong gains, with ablations on the prompt modules and retrieval size.

Critical strengths
  1. The method addresses a practical problem with a sensible idea: use complete neighbors as extra evidence without reconstructing the missing modality, while downweighting noisy labels.
  2. The method is tested on three different tasks, and the ablations show that the two prompt modules work best together.
Critical weaknesses
  1. Table 1 compares RALAP at a 100% missing rate with baselines at 70%, yet the paper still claims 10 of 12 wins. Table 8 uses the same missing rate but reports only the "Both" column and does not establish that the backbone, retriever, and tuning budget are matched. No multi-seed uncertainty is reported.
  2. The paper does not define L_attn or fully specify the memory bank, data splits, query exclusion, retriever, prompt sizes, or trainable parameter count. This makes leakage and reproducibility difficult to assess. The theory also assumes the desired label margin and its positive effect on generation rather than showing that these conditions hold.
Questions for authors
  1. What exactly do Text, Image, and Both mean, and how can Both be evaluated at a 100% missing rate? Please provide all baselines under the same missing patterns, backbone, retriever, and training budget.
  2. How is the memory bank built for each split? Are test samples, duplicate items, and the target's paired modality excluded from retrieval?
  3. Please define L_attn and report prompt dimensions, parameter count, latency, and results over multiple seeds. Do the theorem's coverage and margin conditions hold on the three datasets?
Reviewer_hxSjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a retrieval augmented prompt learning for multimodal classification in the missing modality settings. The model proposes a frozen-LLM framework that transforms retrieved MM neighbors into soft dynamic prompts.

The manuscript evaluates on several benchmarks including MM-IMDB, HateMemes and food 101 and demonstrates good results.

Critical strengths

The paper is overall written well and I was able to follow it reasonably, although at times it is terminology-heavy.

The method is demonstrated to have good benchmark results on several datasets including MM-IMDB, HateMemes and food 101 which are popular datasets for missing modality problem.

Critical weaknesses

The formatting of the paper needs revision. Fonts of the figures is often unreadable and small, especially in Fig 3 and 4.

The evaluation section is critically limited. In the existing literature, missing-modality approaches often evaluate under different levels of missing modalities and settings. The current manuscript only states 'RALAP is evaluated at a 100% missing rate, while the reported baseline results use a 70% missing rate. Therefore, the absolute performance differences should be interpreted with this difference in evaluation setting in mind.' However, I couldnt find any further explanation and analysis on various rates of various missing modalities.

Minor correction: line 085 - RALAP(Fig. 2) -> RALAP (Fig. 2)

Questions for authors

In addition to the concerns raised in critical weakness section, I have the following question from the authors. I do not expect experiments for these answers, only discussion/insights would be sufficient, and maybe appropriate discussion in the paper would be perfect on top of it.

Have the authors considered the problem of corrupt modalities instead of missing? Missing modalities are easy to distinguish and can be handled. However, corrupt modalities are not that easy to identify and solve. For example, see [1].

[1] Multimodal Learning Under Imperfect Data Conditions: A Survey

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.