acceptodds
Under review as a conference paper at ICLR 2027

Dual-Guided Prompt Learning under Missing Modalities

Abstract

Existing prompt-based methods for incomplete multimodal inputs typically use a single fixed set of prompts across all missing patterns, without adapting them to the sample-specific task-relevant evidence provided by the available modalities. Moreover, they cannot effectively recover the complementary semantic information associated with the missing modality for each sample. To address these limitations, we propose Dual-Guided Prompt Learning (DGP), a framework that combines condition-guided prompt adaptation with prototype-guided semantic compensation. For condition guidance, we develop an evidence-aware hierarchical prompt tree, where the missing pattern determines the routing space and a sample-wise Fisher score dynamically selects and aggregates prompt experts. For prototype guidance, we build a learnable cross-modal prototype memory that uses the available modality representation to retrieve the corresponding prototype and inject it into the missing-modality proxy representation. Together, these two forms of guidance enable DGP to perform sample-wise prompt adaptation and compensate the missing branch with retrieved cross-modal semantic propotypes, without reconstructing the raw missing inputs. Experiments on three visual recognition benchmarks show that DGP achieves the best performance in 24 of 27 test configurations. Further evaluation on two multimodal sentiment analysis benchmarks demonstrates its broader applicability, yielding an average accuracy improvement of 2.03 points over the strongest baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal learning, Missing modality

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_a2FPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dual-Guided Prompt Learning (DGP), a framework that combines condition-guided prompt adaptation with prototype-guided semantic compensation. The framework consists of two main components. The first is an evidence-aware hierarchical prompt tree, where the missing pattern determines the routing space and a sample-wise Fisher score dynamically selects and aggregates prompt experts. The second is a learnable cross-modal prototype memory that uses the available modality representation to retrieve the corresponding prototype and inject it into the missing-modality proxy representation. The authors evaluate the proposed method on three visual recognition benchmarks and two multimodal sentiment analysis benchmarks. The experimental results demonstrate competitive performance compared with existing methods.

Critical strengths
  1. The experimental analysis is comprehensive. The authors evaluate the proposed method on five datasets across different tasks and missing-modality settings, demonstrating its effectiveness and generalizability.
  2. The paper is clearly organized and easy to follow. The overall framework and experimental results are presented in a straightforward and structured way, making the proposed approach easy to understand.
Critical weaknesses
  1. Insufficient justification of Fisher information for prompt routing. In Eq. (1), the authors do not specify which network is used to obtain the predictive distribution. And it is unclear whether Fisher information outperforms simpler routing methods, such as prediction entropy or directly using input features.
  2. Limited validation of the claimed cross-modal semantic correspondence. In Section 3.3, the authors use the available modality to retrieve prototypes for the missing modality. However, the paper does not directly show whether these prototypes capture meaningful semantic relationships between modalities.
Questions for authors
  1. In Eq. (1), the predictive distribution cannot be obtained from the input feature (h) directly without any mapping that turn features to class probabilities. The authors should explicitly specify in the paper what kind of network is used. Could the authors clarify how the predictive distribution is computed?
  2. In Section 3.2, the authors use Fisher information to guide prompt routing, but it is unclear whether Fisher information outperforms simpler methods. Could the authors compare it with other routing strategies to demonstrate its advantages?
  3. I noticed that the authors also evaluate the proposed method on two three-modality datasets, CMU-MOSI and CMU-MOSEI, but Eqs. (6)–(10) only describe the interaction between two modalities. Could the authors extend these formulations to the three-modality setting?
3 comments

We thank the reviewer for the insightful comments and suggestions. We address the concerns point by point below.

1. Definition and Justification of Fisher Routing

Thank you for requesting this clarification. A task-specific linear classification head maps the available-modality representation h\mathbf h to logits z=Wh+b\mathbf z=\mathbf W\mathbf h+\mathbf b. The predictive distribution is obtained using softmax for Food-101 and Hateful Memes, and independent sigmoid outputs defining a factorized Bernoulli distribution for MM-IMDb. The Fisher trace with respect to h\mathbf h can be evaluated analytically using these probabilities and classifier weights, without additional backpropagation. We will explicitly specify this mapping in the revised manuscript.

To assess the contribution of Fisher guidance, with identical architectures, training protocols, and parameter budgets, we evaluated three alternative routers on MM-IMDb at a 70% missing rate. These variants route experts using raw available features, preliminary prediction entropy, or the largest predicted sigmoid probability. Fisher routing outperforms the strongest alternative by 1.54, 1.49, and 2.01 F1-Macro points across all three missing patterns. These gains confirm that representation sensitivity captures task evidence better than raw features or output confidence.

RouterMissing-BothMissing-ImageMissing-Text
Feature53.6055.5350.54
Entropy53.2854.3051.52
Maximum confidence53.5555.2151.51
Ours55.1457.0253.53

2. Validation of Cross-Modal Semantic Correspondence

Across the 16 memory slots on MM-IMDb, same-index visual and textual retrieval keys exhibit a mean cosine similarity of 0.849, compared with 0.291 for mismatched pairs. Each key’s nearest neighbor in the other modality is its designated counterpart, demonstrating alignment in the learned retrieval-key space. To assess the functional contribution of cross-modal slot pairing, we trained and evaluated a capacity-matched independent-memory variant with separate retrieval weights, the same number of slots per modality, and pairing-specific alignment losses disabled. We also evaluated unconditional prototype injection by setting g=1g=1 for incomplete samples at inference while keeping the checkpoint, retrieved prototypes, and injection scale fixed. The paired prototype memory outperforms the independent-memory variant, with gains ranging from 1.67 to 2.24 points. Confidence gating further improves performance. These results support the contributions of cross-modal slot pairing and selective semantic compensation.

Variant / Missin-both 70%MM-IMDb (Macro-F1 ↑)Food-101 (Accuracy ↑)Hateful Memes (AUROC ↑)
Independent modality memories53.4783.0165.59
Unconditional prototype injection54.8984.2167.02
Ours55.1485.0767.83

Under the 90% missing rate setting, we examine how prototype learning depends on the amount of paired supervision. Specifically, we reduce Stage I complete training pairs from our default 30% to the exact 10% budget available under this split. Despite using one third of the paired data, DGP drops by at most 0.74 points and still surpasses AOEPT in 8 of 9 configurations. This result confirms that our compact prototype memory captures reliable cross modal correspondences even from scarce complete pairs.

Dataset (Metric)SettingMissing-BothMissing-ImageMissing-Text
MM-IMDb (Macro-F1)DGP52.3555.5051.64
DGP with 10% Stage I52.2055.2150.90
AOEPT50.1354.0950.55
Food101 (Accuracy)DGP82.5887.5278.64
DGP with 10% Stage I82.3087.2878.39
AOEPT81.6787.0377.47
Hateful Memes (AUROC)DGP67.1366.9369.84
DGP with 10% Stage I67.2066.8269.30
AOEPT68.6766.1869.12

3. Extension to Three Modalities

Thank you for pointing this out. For CMU-MOSI and CMU-MOSEI, let R={t,a,v}\mathcal R=\{t,a,v\} denote the text, audio, and video modalities, and let ∅≠A⊆R\emptyset\ne\mathcal A\subseteq\mathcal R denote the available modalities. Following Eq. (6), each shared memory slot ii contains a retrieval key kir\mathbf k_i^r and a semantic prototype Mir\mathbf M_i^r for each modality r∈Rr\in\mathcal R. Each available modality computes its retrieval distribution wr\mathbf w^r as in Eq. (7). We average these distributions over the available modalities:

=\frac{1}{|\mathcal A|} \sum_{r\in\mathcal A}\mathbf w^r.$$ For each missing modality $\bar r\in\mathcal R\setminus\mathcal A$, Eqs. (8)–(10) extend to $$\mathbf S^{\bar r\leftarrow\mathcal A} =\sum_{i=1}^{K}w_i^{\mathcal A}\mathbf M_i^{\bar r},$$ $$g_{\bar r\leftarrow\mathcal A} =\mathbb I_{\bar r\text{-miss}}\, \sigma\!\left( \frac{\max_i w_i^{\mathcal A}-\delta}{\gamma} \right),$$ $$\widetilde{\mathbf h}^{\bar r} =\mathbf h_{\mathrm{miss}}^{\bar r} +g_{\bar r\leftarrow\mathcal A}\, \rho_{\bar r}\!\left( \mathbf S^{\bar r\leftarrow\mathcal A} \right).$$ Here, the missing-modality indicator, gate parameters, and projection retain their definitions in the original formulation. When text and audio are available, their averaged retrieval distribution retrieves the video prototype. When only text is available, its retrieval distribution retrieves the audio and video prototypes separately. For the original bimodal setting with $\mathcal A=\{r\}$, these expressions reduce to Eqs. (8)–(10). We will include this extension in the revised manuscript.
Reviewer_K72JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dual-Guided Prompt Learning (DGP) for multimodal learning with missing modalities. A hierarchical prompt tree first selects a branch from the missing pattern, then uses a sample-wise Fisher score to mix three prompt experts. A paired cross-modal prototype memory, learned from complete samples, retrieves a semantic proxy for the missing modality. The model is trained in two stages. DGP obtains the best result in 24 of 27 visual-recognition settings and improves average accuracy over the strongest reported sentiment baselines by 2.28 points on CMU-MOSI and 1.77 points on CMU-MOSEI. Ablations support contributions from routing and prototype compensation.

Critical strengths
  1. The design addresses two practical limitations of pattern-specific prompting: samples with the same missing pattern can receive different prompts, and the missing branch receives compact cross-modal semantic information rather than a reconstructed raw input.
  2. The experiments are comprehensive and cover five datasets and various missing-modality settings.
Critical weaknesses
  1. The novelty relative to the closest prior work would benefit from a clearer comparison. Semantically Conditioned Prompts for Visual Recognition under Missing Modality Scenarios (WACV 2025) already generates sample-specific prompts from available-modality semantics using a learnable memory bank, and AOEPT also produces instance-aware prompts. The paper does not discuss the WACV work or compare Fisher routing with simpler confidence-, entropy-, or feature-based routers, so the specific benefit of the proposed routing signal is not yet fully isolated.
  2. The empirical conclusions need stronger uncertainty and cost reporting. Many Table 1 margins are between 0.02 and 0.75 points, while DGP itself has standard deviations of 0.26 to 0.53; the baselines do not have matched multi-seed results or paired tests. DGP also uses 91.61 GFLOPs versus 44.32 to 53.32 for the reported prompt baselines, and the table does not clearly account for the per-sample gradient computation required by the Fisher score at inference.
Questions for authors
  1. How does DGP differ technically and empirically from Semantically Conditioned Prompts (WACV 2025) and AOEPT? Please also compare the Fisher-score router with entropy, maximum confidence, and a router that directly uses the available feature.
  2. Do the 24-of-27 visual-recognition conclusions remain after running DGP and the strongest baselines with the same seeds and reporting paired confidence intervals or significance tests?
  3. How is the Fisher trace computed during inference, and do the reported GFLOPs include its gradient calculation? Wall-clock latency and memory would clarify the practical overhead.
  4. Since Stage I uses complete image-text pairs, how does performance change as the proportion of complete training pairs approaches zero?
5 comments

We thank the reviewer for the careful reading. We address the concerns point by point below, and we will also incorporate the corresponding revisions into the manuscript.

1. Technical Analysis and Justification of Fisher Routing

Thank you for pointing out the omissions in our analysis of existing work. Both SCP and AOEPT generate instance-specific prompts conditioned on available modalities to improve robustness to missing modalities. However, compensation through prompts relies on attention interactions to convey complementary semantics, without explicitly retrieving a matched counterpart for the missing modality. DGP instead uses a learnable paired cross-modal semantic prototype memory. Available modality features retrieve the corresponding missing-modality prototype, which is injected into the missing-modality proxy representation through a confidence-controlled gate. Combined with the Fisher-guided hierarchical prompt tree, DGP provides dual guidance for sample-wise adaptation and explicit semantic compensation.

We further examined replacing the Fisher guidance with three alternative routing criteria under identical architectures, training settings, and parameter counts. Specifically, we evaluated a feature router, predictive entropy, and maximum prediction confidence on MM-IMDb at a 70% missing rate. Across all three missing patterns, Fisher routing exceeds the strongest baseline router by 1.54, 1.49, and 2.01 F1-Macro points, showing that representation sensitivity captures task evidence more effectively than unconditioned features or scalar output confidence.

RouterMissing-BothMissing-ImageMissing-Text
Feature53.6055.5350.54
Entropy53.2854.3051.52
Maximum confidence53.5555.2151.51
Ours55.1457.0253.53

2. Reliability of Performance Gains

Appendix B.2 reports DGP’s variability across three random seeds under the 70% missing-both setting, but this alone does not quantify uncertainty in its improvements over the baselines. We will extend the evaluation to DGP and the strongest baselines using matched seeds, identical data splits, and shared missing-modality masks, and report performance differences. Across all three random seeds, DGP achieves the highest mean performance in 25 of the 27 configurations, surpassing both AOEPT and DPL in every setting on MM-IMDb and Food-101. DGP also exhibits tight standard deviations between 0.04 and 0.44 on MM-IMDb and Food-101, and between 0.31 and 0.75 on Hateful Memes, matching or improving upon the stability of both baselines. Moreover, the performance margins of DGP exceed the corresponding standard deviations across most settings, confirming that its gains remain consistent under matched random seeds.

DatasetsMissing configurationsAOEPTDPLOurs
Missing-Text/Image/Both 50%53.88 ± 0.10 / 56.04 ± 0.14 / 54.69 ± 0.1955.13 ± 0.27 / 57.89 ± 0.35 / 56.08 ± 0.2755.71 ± 0.34 / 58.62 ± 0.27 / 57.31 ± 0.21
MM-IMDbMissing-Text/Image/Both 70%51.61 ± 0.17 / 54.41 ± 0.48 / 53.04 ± 0.2753.03 ± 0.30 / 56.68 ± 0.40 / 55.10 ± 0.5153.63 ± 0.20 / 57.16 ± 0.44 / 55.20 ± 0.37
Missing-Text/Image/Both 90%50.59 ± 0.21 / 53.91 ± 0.34 / 50.39 ± 0.3350.91 ± 0.12 / 54.63 ± 0.38 / 52.12 ± 0.3251.83 ± 0.22 / 55.63 ± 0.30 / 52.68 ± 0.38
Missing-Text/Image/Both 50%84.28 ± 0.23 / 89.45 ± 0.16 / 86.08 ± 0.2584.21 ± 0.10 / 89.58 ± 0.12 / 86.61 ± 0.1385.66 ± 0.19 / 89.89 ± 0.15 / 87.79 ± 0.11
Food-101Missing-Text/Image/Both 70%80.71 ± 0.19 / 88.80 ± 0.19 / 83.27 ± 0.1581.28 ± 0.29 / 87.84 ± 0.09 / 83.36 ± 0.2382.22 ± 0.12 / 88.99 ± 0.04 / 85.05 ± 0.26
Missing-Text/Image/Both 90%77.52 ± 0.09 / 87.08 ± 0.14 / 81.73 ± 0.2178.67 ± 0.17 / 86.24 ± 0.07 / 80.99 ± 0.1878.76 ± 0.13 / 87.47 ± 0.16 / 82.60 ± 0.16
Missing-Text/Image/Both 50%69.53 ± 0.60 / 66.55 ± 0.17 / 68.13 ± 0.6969.77 ± 1.18 / 65.69 ± 0.60 / 64.41 ± 0.7271.54 ± 0.58 / 68.52 ± 0.42 / 69.01 ± 0.31
Hateful MemesMissing-Text/Image/Both 70%69.09 ± 0.58 / 66.52 ± 0.61 / 69.61 ± 0.7169.07 ± 0.75 / 64.03 ± 0.46 / 63.25 ± 0.8670.84 ± 0.45 / 67.31 ± 0.56 / 67.90 ± 0.53
Missing-Text/Image/Both 90%68.89 ± 0.47 / 66.05 ± 0.22 / 67.93 ± 0.7569.04 ± 0.60 / 63.26 ± 0.77 / 64.33 ± 0.8769.89 ± 0.47 / 66.96 ± 0.66 / 67.54 ± 0.75

3. Fisher Score Computation and Full Inference Cost

With a linear classification head, the Fisher trace with respect to h\mathbf h can be evaluated analytically using predicted probabilities and classifier weights, without additional backpropagation. The 91.61 GFLOPs figure in Table 7 reflects the full inference workflow under an identical backbone and input setup. It covers preliminary feature extraction and prediction, Fisher score evaluation, prototype retrieval, and the final prompted prediction. Data loading and preprocessing times are excluded from all measurements. Because our preliminary classifier uses a linear head, the Fisher trace is calculated analytically from predicted probabilities and classifier weights. We have revised Appendix B.1 to make this measurement boundary explicit. We also added end to end latency and peak GPU memory comparisons. All models are evaluated under the same hardware, numerical precision, and a batch size of 1. DGP runs at 28.63 ms per sample and consumes only 0.610 GiB of peak GPU memory. It adds just 4.24 ms over AOEPT, runs 29.5% faster than RAGPT, and matches the lightweight memory footprint of DCP and AOEPT. Overall, our two stage framework delivers substantial performance gains across 24 of 27 visual settings without significantly increasing inference latency or memory overhead.

MethodEnd-to-end latency (ms/sample) ↓Peak GPU memory (GB) ↓
DCP15.550.598
AOEPT24.390.584
RAGPT40.602.910
Ours28.630.610
Reviewer_PsP5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Dual-Guided Prompt Learning (DGP) for classification with missing modalities. A hierarchical prompt tree first selects a branch according to the missing pattern, then uses a sample-level Fisher score to combine prompt experts. A paired prototype memory, learned from complete training examples, retrieves features for the missing modality from the observed one. Experiments cover MM-IMDb, Food101, Hateful Memes, CMU-MOSI, and CMU-MOSEI. DGP obtains the highest reported score in 24 of the 27 image-text settings in Table 1, including all nine Food101 settings. The paper also examines changes in missing rates, component ablations, computational cost, and variation across three random seeds.

Critical strengths
  1. Adapting prompts within a missing-pattern group is well motivated. Two samples can have the same available modalities but differ in how informative those observations are. DGP allows them to use different mixtures of prompt experts. This addresses a limitation of prompts selected only by the missing pattern.
  2. The Food101 results are consistent across the nine settings in Table 1 and the changes in test missing rate in Figure 2. Table 4 further shows gains over both no compensation and an instance memory bank built from the same complete training data. These comparisons give concrete support for the proposed adaptation and compensation modules.
Critical weaknesses
  1. The training protocol does not make the amount of complete paired data sufficiently clear. Section 3.4 and Appendix A.1 describe learning the memory from complete examples before simulating missing inputs in Stage II, whereas Section 4 describes missingness during both training and testing. It is unclear whether Stage I uses all original training pairs or only the complete subset allowed by each missing-rate setting. This distinction matters especially at 90% missingness. Table 4 controls data access for the instance-memory ablation, but the corresponding access for the methods in Table 1 is not explicit. I cannot judge whether the main comparisons use equivalent training information without this clarification.
  2. The ablations support the modules but do not yet establish the need for their specific design. Table 3 tests removal of evidence guidance and adaptive routing; it does not compare the Fisher score with a learned feature router or a simpler confidence measure. Table 4 includes an instance-memory control, but it does not isolate the cross-modal slot pairing or the confidence gate. These distinctions matter because DGP requires 91.61 GFLOPs, compared with 44.32 for AOEPT and 53.32 for RAGPT (Table 7). Appendix B.1 describes the measurement as one forward pass, leaving it unclear whether the gradient computation for Fisher routing is included. I would like stronger evidence that the proposed mechanisms justify this additional cost.
Questions for authors
  1. Which complete pairs are used in Stage I at each missing rate? Please give the number of pairs and explain whether the baseline methods have access to the same information before training-time masking.
  2. How is the predictive distribution in Eq. (1) defined for MM-IMDb's multilabel outputs, and how is the expectation evaluated in practice? A comparison with a feature-based router of similar capacity would help establish whether the Fisher score is necessary.
  3. Does Table 7 include the preliminary prediction and the gradient computation needed for the Fisher score, as well as the final prompted prediction? Please clarify the measurement scope and provide end-to-end latency under the same hardware and batch size as the baselines.
6 comments

We thank the reviewer for the insightful comments. We address the concerns point by point below.

1. Training Data Protocol and Fairness of Comparison

Thank you for raising this concern about the fairness of training data protocol. By default, we randomly select complete image–text pairs corresponding to 30% of the training set for Stage I. We acknowledge that this allocation exceeds the complete-pair budget permitted under the 90% missing-rate setting. We have therefore restricted Stage I to the 10% complete training pairs available in this setting, matching the complete-pair budget of the baselines. Even with Stage I restricted to the 10% complete-pair budget, DGP retains nearly identical performance and still outperforms AOEPT in 8 of the 9 settings, showing that a compact cross-modal prototype memory can be effectively learned from limited complete pairs.

Dataset (Metric)SettingMissing-BothMissing-ImageMissing-Text
MM-IMDb (Macro-F1)DGP52.3555.5051.64
DGP with 10% Stage I52.2055.2150.90
AOEPT50.1354.0950.55
Food101 (Accuracy)DGP82.5887.5278.64
DGP with 10% Stage I82.3087.2878.39
AOEPT81.6787.0377.47
Hateful Memes (AUROC)DGP67.1366.9369.84
DGP with 10% Stage I67.2066.8269.30
AOEPT68.6766.1869.12

2. Fisher Score Definition and Implementation

Thank you for pointing this out. For MM-IMDb, a linear classification head maps the available-modality representation h\mathbf{h} to logits z=Wh+b\mathbf{z}=\mathbf{W}\mathbf{h}+\mathbf{b}. With qc=σ(zc)q_c=\sigma(z_c), we define the multilabel predictive distribution as

=\prod_{c=1}^{C}q_c^{y_c}(1-q_c)^{1-y_c}.$$ Under this conditionally independent Bernoulli model, the expectation in Eq. (1) is evaluated analytically: $$\operatorname{Tr}\mathcal{I}(\mathbf{h}) =\sum_{c=1}^{C}q_c(1-q_c)\|\mathbf{w}_c\|_2^2,$$ since $\nabla_{\mathbf{h}}z_c=\mathbf{w}_c$. This computation requires no ground-truth labels, Monte Carlo sampling, or additional backpropagation. We will clarify these details in the revised manuscript.

3. Justification of Fisher Routing and the Memory Design

To isolate the effect of the routing signal, we evaluate three alternatives to Fisher routing on MM-IMDb at a 70% missing rate. The feature-based router uses the available features directly, the entropy-based router uses the entropy of the preliminary predictions, and the maximum-confidence router uses the largest predicted sigmoid probability across labels. We match the number of trainable routing parameters and keep the rest of the model architecture and the missing protocol fixed. Fisher-guided routing outperforms the strongest alternative by 1.54, 1.49, and 2.01 F1-Macro points across the three missing patterns, showing that local predictive sensitivity with respect to the available representation provides a more informative routing signal than raw features or output-level confidence alone.

RouterMissing-BothMissing-ImageMissing-Text
Feature53.6055.5350.54
Entropy53.2854.3051.52
Maximum confidence53.5555.2151.51
Ours55.1457.0253.53

Using the trained DGP model on MM-IMDb, we computed the cosine similarity between all image and text retrieval keys across the 16 memory slots. The mean similarity was 0.849 for same index pairs and 0.291 for the mismatched pairs. In both directions, all 16 keys had their designated counterpart as the nearest key in the other modality. These results show that the paired slots are aligned in the learned retrieval key space.

To isolate the effect of confidence gating at inference, we evaluated unconditional prototype injection by setting g=1g=1 for incomplete samples while holding the checkpoint, retrieved prototypes, and injection scale fixed. To assess cross-modal slot pairing, we trained and evaluated a capacity-matched independent-memory variant. The two modality banks use separate retrieval weights, with the same number of slots per modality and the pairing-specific alignment losses disabled. Paired cross-modal slot alignment yields gains of 1.67 to 2.24 points over independent memories by ensuring semantically consistent cross-modal retrieval, while confidence gating provides an additional 0.25 to 0.86 points by suppressing unreliable prototype injections.

Variant / Missin-both 70%MM-IMDb (Macro-F1 ↑)Food-101 (Accuracy ↑)Hateful Memes (AUROC ↑)
Independent modality memories53.4783.0165.59
Unconditional prototype injection54.8984.2167.02
Ours55.1485.0767.83

4. Full Inference Cost and Performance Gains

Thank you for highlighting this ambiguity. The 91.61 GFLOPs reported in Table 7 covers the complete DGP inference pipeline, including preliminary feature extraction and prediction, the full Fisher-score computation, prototype retrieval, and the final prompted prediction. All methods use the same backbone and input configuration. We will revise the wording in Appendix B.1 to make this measurement scope explicit. We further measured end to end inference latency and peak GPU memory. All models were tested under identical GPU hardware, numerical precision, and a batch size of 1. Despite its two stage procedure, DGP delivers substantial performance gains with moderate inference overhead. It runs at 28.63 ms per sample, adding only 4.24 ms over AOEPT while reducing latency by 29.5% compared with RAGPT. Meanwhile, its peak GPU memory remains at 0.610 GiB, matching the lightweight footprint of DCP and AOEPT for a practical balance between performance and inference cost.

MethodEnd-to-end latency (ms/sample) ↓Peak GPU memory (GiB) ↓
DCP15.550.598
AOEPT24.390.584
RAGPT40.602.910
Ours28.630.610

Thank you for the detailed response and the additional experiments. I have two remaining questions about the memory design.

  1. The independent-memory variant performs worse than using no memory at all. Under the missing-both 70% setting, the no-memory ablation in Table 5 scores 54.54 F1-Macro on MM-IMDb, 83.94 accuracy on Food101, and 66.75 AUROC on Hateful Memes. The independent-memory variant in your response scores 53.47, 83.01, and 65.59 on the same three datasets, which is 1.07, 0.93, and 1.16 points lower. In this control, adding a memory therefore hurts the model.

The reported gain of 1.67 to 2.24 points from paired slot alignment is measured against this weakened control. Measured against the no-memory variant, the full model (55.14, 85.07, and 67.83) gains 0.60, 1.13, and 1.08 points, about half the reported figure.

This raises the question of what the control is testing. Could you clarify how, in this variant, the available modality retrieves a prototype for the missing modality when the two banks have separate retrieval weights and no shared slot index? If the injected prototype is unrelated to the input, the result shows that injecting unrelated content is harmful, which is a different conclusion from the need for paired slots.

  1. The key-similarity analysis measures a property that the training objective enforces. The response reports a mean cosine similarity of 0.849 between same-index image and text keys, 0.291 between mismatched keys, and that every key has its designated counterpart as the nearest neighbor. It then attributes the performance gain to "semantically consistent cross-modal retrieval".

These statistics follow directly from the Stage I losses. LpairL_{pair} (Eq. 16) penalizes every slot whose same-index key similarity is below 1−mp=0.801 − m_p = 0.80, so minimizing it yields a value just above 0.80. LdivL_{div} (Eq. 17) pushes apart keys from different slots within each modality, which, combined with LpairL_{pair}, keeps mismatched cross-modal similarities low. The nearest-neighbor result follows from these two. The same numbers would therefore be obtained whether or not the pairing helps the task, so they show that the losses were optimized and cannot serve as evidence that the retrieval is semantically meaningful.

If the two points above are answered with well-supported explanations, I would be willing to raise my rating from 2 to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.