acceptodds
Withdrawn from ICLR 2027

Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large-scale multimodal models (LMMs) have achieved superior performance in visual recognition by synergizing information across diverse, massive-scale paired modalities. In real-world scenarios, however, missing-modality inputs are ubiquitous, causing models optimized for modality-complete data to exhibit precipitous performance degradation. Existing research has introduced prompt learning to mitigate this issue, typically by generating dynamic prompts from instance-level features, regardless of whether the input modalities are complete or partially absent. However, such input-conditioned strategies are hindered by the escalating unreliability of instance-level features; as higher missing rates increase the proportion of incomplete modalities, the resulting instability in prompt learning limits the model's performance. To address this limitation, we hypothesize that learnable latent prompts themselves encapsulate stable, modality-intrinsic priors that are decoupled from corrupted inputs. Consequently, we propose a novel paradigm: Learning from Reliable Latent Prompts. Unlike prior methods, we model input-agnostic learnable prompts as stable latent anchors that enable robust guidance and effective cross-modal knowledge compensation, even under extreme missing rates (e.g., 90%). Empirical results across three benchmark datasets demonstrate that our "learn-from-latent-prompts" approach achieves state-of-the-art performance across a wide range of missing-modality scenarios. Extensive experiments further confirm the effectiveness of this paradigm in providing a robust solution to the missing-modality problem. Code will be released after acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Missing Modalities, Prompt Learning, Visual Recognition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cqH2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Learning from Reliable Latent Prompts, a prompt-learning framework for visual recognition under missing-modality scenarios. The main motivation behind this work is that existing input-conditioned prompting methods rely on potentially incomplete or unreliable modality features, which can negatively affect prompt learning. The proposed method introduces input-independent, modality-specific learnable prompts that are transformed into latent anchors. A cross-modal co-adaptation mechanism uses bidirectional cross-attention between these anchors to generate dual anchor-induced prompts, which are concatenated with the original anchors and injected into the frozen CLIP image and text encoders. The method is evaluated on different image-text benchmarks under various missing-modality scenarios, and the authors report improvements over the selected baselines. Ablation studies are also provided to evaluate the effectiveness of the proposed components.

Critical strengths
  1. The paper shows strong experimental performance, as the proposed method outperforms the selected baselines across different missing-modality scenarios.

  2. The proposed method is relatively simple and does not require explicit reconstruction of missing modalities.

  3. The paper is well written, organized, and easy to follow.

Critical weaknesses
  1. The proposed method relies on input-independent latent anchors that are shared across samples and are not conditioned on the available modality content. Although these anchors interact with the actual input features through self-attention within the CLIP encoders, the cross-modal co-adaptation mechanism itself operates only on the learned anchors and does not adapt the generated prompts to instance-specific semantics. While the authors claim that these anchors provide semantic compensation for missing modalities, it remains unclear how input-independent prompts can compensate for different missing information across samples. Therefore, it is difficult to determine whether the reported improvements result from meaningful cross-modal semantic compensation or general prompt-based adaptation. The provided experiments and ablation studies do not directly validate this claim or establish whether input-independent anchors are preferable to prompts conditioned on the available modality features.

  2. The paper does not evaluate the performance of the proposed method under a 100% missing rate, where all samples have only one available modality (either text or image). Given the authors' argument that input-independent latent anchors learn stable modality-level priors that can compensate for missing information, it would be important to evaluate how the method performs when an entire modality is unavailable across all samples.

  3. The paper is missing comparisons with several relevant methods that address the same missing-modality problem. Some of the works that are not included are:

  • Lu et al., "DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities," CVPR, 2026

  • Zhao et al., "Enhancing Multimodal Model Robustness Under Missing Modalities via Memory-Driven Prompt Learning," IJCAI, 2025

  • Pipoli et al., "Semantically Conditioned Prompts for Visual Recognition under Missing Modality Scenarios," WACV, 2025

  • Nezakati et al., "MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection," IEEE BigData, 2025

  • Reza et al., "Robust Multimodal Learning via Cross-Modal Proxy Tokens," TMLR, 2025

Questions for authors
  1. The authors claim that the learned anchors capture meaningful modality-specific information, but how is this demonstrated? It is unclear whether these anchors actually encode information specific to each modality or simply function as generic learned prompts that improve classification performance. What evidence supports the modality-specific nature of these anchors?

  2. The authors argue that input-independent prompts are more reliable than input-conditioned prompts. However, when one modality is missing, the other modality is still available and contains useful semantic information. What justifies not using this information during prompt generation? The current comparisons do not clearly establish whether input-independent anchors are more effective than using the available modality features.

  3. The experiments focus on classification tasks using image-text datasets. How well would the proposed method generalize to other downstream tasks, such as visual question answering, if the learned anchors in fact capture meaningful modality-specific information? The current evaluation does not establish whether these anchors would remain effective beyond classification.

Reviewer_MF4XOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Learning from Reliable Latent Prompts (LLP), a prompting framework that adopts a latent prior-driven paradigm, decoupling prompt learning from unreliable inputs. Experiments show its effectiveness.

Critical strengths

The tackled problem, modality-missing learning in multimodal transformers, is meaningful.

Critical weaknesses
  1. Compared to AOEPT (ICML 2026), which constructs global modal priors through training samples and performs instance-level adaptation, LLP implicitly learns modal priors by only optimizing input-irrelevant learnable prompts, while discarding instance-level conditioning mechanisms. The authors do not fully explain why this simpler design, lacking instance adaptation capabilities, can provide more reliable and effective missing modal compensation. Meanwhile, the novelty, is also not sufficient considering the AOEPT.

  2. Missing Key Baselines, SOTA Claims Lack Support. The paper omits highly relevant and important works such as AOEPT and RAGPT (AAAI 2025), which have explored the use of modal priors or retrieved knowledge for missing modal compensation, further undermining LLP's claims of novelty. More importantly, under multiple similar experimental setups, LLP's reported results are even inferior to AOEPT, rendering its SOTA claims unsupported.

Questions for authors

See the weaknesses, and also think about the below question:

Q: LLP's CMCA performs cross-attention only on input-irrelevant static anchors without leveraging the image or text features of the current sample. Additionally, the two CLIP encoders extract features independently and fuse them only through the final classification layer. Consequently, this mechanism is closer to reparameterizing static prompts rather than true instance-level cross-modal interaction, and its claimed cross-modal semantic compensation lacks sufficient mechanism support.

Reviewer_vHhnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses multimodal classification under missing image or text inputs. It proposes Learning from Reliable Latent Prompts (LLP), which introduces modality-specific, input-agnostic latent anchors and a cross-modal co-adaptation (CMCA) mechanism to generate refined prompts. Experiments on three benchmarks demonstrate improved performance across different missing-modality settings, supported by ablation, robustness, and generalization analyses.

Critical strengths
  1. Practical relevance: The paper addresses the important problem of missing modalities in multimodal recognition and proposes a relatively simple framework without fine-tuning the pretrained backbone.

  2. Extensive experimental validation: The method consistently outperforms existing approaches across three benchmarks and multiple missing-modality settings. The ablation studies, cross-backbone evaluation, and stability analyses further support its effectiveness.

Critical weaknesses
  1. Limited methodological novelty: The proposed framework largely combines existing techniques, including learnable static prompts, cross-modal attention, and layer-wise prompt updates. The latent anchors are conceptually similar to conventional static prompts, while cross-modal prompt interaction has also been explored in prior work. Therefore, the claimed novelty of the learning paradigm is not sufficiently established.

  2. Insufficient justification of the core mechanism: Both the latent anchors and their cross-modal interactions are input-independent. It is therefore unclear how CMCA achieves the claimed semantic compensation for missing modalities at the instance level. Although the experiments demonstrate performance improvements, they do not directly establish the proposed compensation mechanism.

Questions for authors

What fundamentally distinguishes the proposed latent anchors from conventional modality-specific static prompts? Could the authors provide more controlled comparisons with existing cross-modal prompt learning methods?

Since CMCA operates entirely on input-independent anchors, how does it compensate for missing instance-specific information? Can the authors provide more direct evidence supporting this claim?

Could the authors report the additional parameters and computational overhead introduced by LLP compared with DCP and SyP?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.