- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Dual-Guided Prompt Learning (DGP), a framework that combines condition-guided prompt adaptation with prototype-guided semantic compensation. The framework consists of two main components. The first is an evidence-aware hierarchical prompt tree, where the missing pattern determines the routing space and a sample-wise Fisher score dynamically selects and aggregates prompt experts. The second is a learnable cross-modal prototype memory that uses the available modality representation to retrieve the corresponding prototype and inject it into the missing-modality proxy representation. The authors evaluate the proposed method on three visual recognition benchmarks and two multimodal sentiment analysis benchmarks. The experimental results demonstrate competitive performance compared with existing methods.
Critical strengths
- The experimental analysis is comprehensive. The authors evaluate the proposed method on five datasets across different tasks and missing-modality settings, demonstrating its effectiveness and generalizability.
- The paper is clearly organized and easy to follow. The overall framework and experimental results are presented in a straightforward and structured way, making the proposed approach easy to understand.
Critical weaknesses
- Insufficient justification of Fisher information for prompt routing. In Eq. (1), the authors do not specify which network is used to obtain the predictive distribution. And it is unclear whether Fisher information outperforms simpler routing methods, such as prediction entropy or directly using input features.
- Limited validation of the claimed cross-modal semantic correspondence. In Section 3.3, the authors use the available modality to retrieve prototypes for the missing modality. However, the paper does not directly show whether these prototypes capture meaningful semantic relationships between modalities.
Questions for authors
- In Eq. (1), the predictive distribution cannot be obtained from the input feature (h) directly without any mapping that turn features to class probabilities. The authors should explicitly specify in the paper what kind of network is used. Could the authors clarify how the predictive distribution is computed?
- In Section 3.2, the authors use Fisher information to guide prompt routing, but it is unclear whether Fisher information outperforms simpler methods. Could the authors compare it with other routing strategies to demonstrate its advantages?
- I noticed that the authors also evaluate the proposed method on two three-modality datasets, CMU-MOSI and CMU-MOSEI, but Eqs. (6)–(10) only describe the interaction between two modalities. Could the authors extend these formulations to the three-modality setting?
3 comments
2. Validation of Cross-Modal Semantic Correspondence
Across the 16 memory slots on MM-IMDb, same-index visual and textual retrieval keys exhibit a mean cosine similarity of 0.849, compared with 0.291 for mismatched pairs. Each key’s nearest neighbor in the other modality is its designated counterpart, demonstrating alignment in the learned retrieval-key space. To assess the functional contribution of cross-modal slot pairing, we trained and evaluated a capacity-matched independent-memory variant with separate retrieval weights, the same number of slots per modality, and pairing-specific alignment losses disabled. We also evaluated unconditional prototype injection by setting for incomplete samples at inference while keeping the checkpoint, retrieved prototypes, and injection scale fixed. The paired prototype memory outperforms the independent-memory variant, with gains ranging from 1.67 to 2.24 points. Confidence gating further improves performance. These results support the contributions of cross-modal slot pairing and selective semantic compensation.
| Variant / Missin-both 70% | MM-IMDb (Macro-F1 ↑) | Food-101 (Accuracy ↑) | Hateful Memes (AUROC ↑) |
|---|---|---|---|
| Independent modality memories | 53.47 | 83.01 | 65.59 |
| Unconditional prototype injection | 54.89 | 84.21 | 67.02 |
| Ours | 55.14 | 85.07 | 67.83 |
Under the 90% missing rate setting, we examine how prototype learning depends on the amount of paired supervision. Specifically, we reduce Stage I complete training pairs from our default 30% to the exact 10% budget available under this split. Despite using one third of the paired data, DGP drops by at most 0.74 points and still surpasses AOEPT in 8 of 9 configurations. This result confirms that our compact prototype memory captures reliable cross modal correspondences even from scarce complete pairs.
| Dataset (Metric) | Setting | Missing-Both | Missing-Image | Missing-Text |
|---|---|---|---|---|
| MM-IMDb (Macro-F1) | DGP | 52.35 | 55.50 | 51.64 |
| DGP with 10% Stage I | 52.20 | 55.21 | 50.90 | |
| AOEPT | 50.13 | 54.09 | 50.55 | |
| Food101 (Accuracy) | DGP | 82.58 | 87.52 | 78.64 |
| DGP with 10% Stage I | 82.30 | 87.28 | 78.39 | |
| AOEPT | 81.67 | 87.03 | 77.47 | |
| Hateful Memes (AUROC) | DGP | 67.13 | 66.93 | 69.84 |
| DGP with 10% Stage I | 67.20 | 66.82 | 69.30 | |
| AOEPT | 68.67 | 66.18 | 69.12 |
3. Extension to Three Modalities
Thank you for pointing this out. For CMU-MOSI and CMU-MOSEI, let denote the text, audio, and video modalities, and let denote the available modalities. Following Eq. (6), each shared memory slot contains a retrieval key and a semantic prototype for each modality . Each available modality computes its retrieval distribution as in Eq. (7). We average these distributions over the available modalities:
=\frac{1}{|\mathcal A|} \sum_{r\in\mathcal A}\mathbf w^r.$$ For each missing modality $\bar r\in\mathcal R\setminus\mathcal A$, Eqs. (8)–(10) extend to $$\mathbf S^{\bar r\leftarrow\mathcal A} =\sum_{i=1}^{K}w_i^{\mathcal A}\mathbf M_i^{\bar r},$$ $$g_{\bar r\leftarrow\mathcal A} =\mathbb I_{\bar r\text{-miss}}\, \sigma\!\left( \frac{\max_i w_i^{\mathcal A}-\delta}{\gamma} \right),$$ $$\widetilde{\mathbf h}^{\bar r} =\mathbf h_{\mathrm{miss}}^{\bar r} +g_{\bar r\leftarrow\mathcal A}\, \rho_{\bar r}\!\left( \mathbf S^{\bar r\leftarrow\mathcal A} \right).$$ Here, the missing-modality indicator, gate parameters, and projection retain their definitions in the original formulation. When text and audio are available, their averaged retrieval distribution retrieves the video prototype. When only text is available, its retrieval distribution retrieves the audio and video prototypes separately. For the original bimodal setting with $\mathcal A=\{r\}$, these expressions reduce to Eqs. (8)–(10). We will include this extension in the revised manuscript.
We thank the reviewer for the insightful comments and suggestions. We address the concerns point by point below.
1. Definition and Justification of Fisher Routing
Thank you for requesting this clarification. A task-specific linear classification head maps the available-modality representation h to logits z=Wh+b. The predictive distribution is obtained using softmax for Food-101 and Hateful Memes, and independent sigmoid outputs defining a factorized Bernoulli distribution for MM-IMDb. The Fisher trace with respect to h can be evaluated analytically using these probabilities and classifier weights, without additional backpropagation. We will explicitly specify this mapping in the revised manuscript.
To assess the contribution of Fisher guidance, with identical architectures, training protocols, and parameter budgets, we evaluated three alternative routers on MM-IMDb at a 70% missing rate. These variants route experts using raw available features, preliminary prediction entropy, or the largest predicted sigmoid probability. Fisher routing outperforms the strongest alternative by 1.54, 1.49, and 2.01 F1-Macro points across all three missing patterns. These gains confirm that representation sensitivity captures task evidence better than raw features or output confidence.