acceptodds
Under review as a conference paper at ICLR 2027

Privileged Posterior Calibration of Memory Routing for Incomplete Multimodal Learning

Abstract

Multimodal learning in real-world settings often faces incomplete inputs, requiring models to reason from only partially observed evidence. Recent prompt-based methods improve robustness by augmenting pretrained multimodal transformers with learnable prompts, while memory-based variants further retrieve instance-specific prompts to compensate for missing information. However, their retrieval decisions are typically conditioned only on the available modalities, making reliable memory access difficult when informative modalities are absent. We study the common setting in which complete multimodal pairs are available during training to simulate missing observations, while inference may receive only partial inputs. We propose Privileged Posterior Calibration (PPC), which uses the unmasked multimodal input as a training-only privileged reference for memory routing. PPC transfers complete-view routing distributions to an observed-view router through confidence-weighted dense calibration, while retaining sparse top-k retrieval for prompt injection and inference. Rather than requiring recovery of an unknowable complete-view route for every input, a conditional-projection analysis shows that the population objective learns the teacher routing predictable from observed evidence. Complete-view prototype initialization and semantic-neighborhood regularization further organize the shared prompt memory. Across MM-IMDb, Food101, and Hateful Memes under the RAGPT/ANGA-aligned 70% missing-rate protocol, PPC consistently outperforms our reproduced MemPrompt baseline in all nine settings. On image-missing MM-IMDb, PPC reaches 51.5 test F1-Macro, compared with 49.3 for complete-view feature distillation and 47.9 for an architecture-matched no-calibration control. These results show that complete-view supervision can improve memory-access decisions beyond alternative uses of privileged training information, while requiring only partial inputs and sparse memory retrieval at inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
incomplete multimodal learning, multimodal transformers, missing modalities, prompt learning, memory-augmented learning, memory routing, privileged information, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hXZzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies memory routing for incomplete multimodal learning, aiming to address the challenge of modality missing at deployment. To this end, the paper proposes Privileged Posterior Calibration (PPC), which leverages complete image-text pairs available during training to supervise how inputs with missing modalities retrieve prompts from the prompt memory. The experimental results appear promising.

Critical strengths
  1. The paper proposes a method for handling missing modalities at test time.

  2. The paper provides theoretical analysis of the proposed method.

  3. The experimental setup is relatively comprehensive, with sufficient baseline comparisons and competitive performance.

Critical weaknesses
  1. Compared with existing methods, the motivation and novelty of the proposed approach are not sufficiently compelling. In addition, the paper is not clearly written.

  2. The experimental evaluation is substantially insufficient. The paper only evaluates a 70% missing rate, rather than covering a broader range of missing rates (e.g., 10%–90%) as in prior work. Moreover, the evaluation relies solely on performance metrics and lacks complementary analyses such as visualizations.

Questions for authors
  1. What are the key differences between the proposed method and existing approaches, and which limitations of prior methods does it aim to address? The paper lacks sufficient discussion of the limitations of existing methods.

  2. Eq. 11 appears to have a potential division-by-zero issue.

  3. Although the paper claims to perform “posterior calibration,” it does not report calibration-specific evaluations such as calibration error, ECE, or reliability analysis.

  4. Prior work such as ANGA evaluates performance across missing rates ranging from 10% to 90%, whereas this paper only considers a 70% missing rate. Results under additional missing rates should be provided for a more comprehensive comparison. Moreover, beyond quantitative performance metrics, the paper would benefit from relevant visualization experiments to provide more complete empirical evidence.

  5. The heavy use of terminology makes the paper difficult to follow. In the abstract, the descriptions of the motivation and method are overly compressed, whereas the discussion of experimental results is disproportionately detailed. In addition, the notation should more clearly distinguish scalars from vectors, and the layouts of Tables 2 and 4 should be improved for better readability.

2 comments

We thank the reviewer for the careful evaluation and constructive suggestions. We appreciate the recognition of our theoretical analysis and experimental comparisons. We have revised the manuscript and added experiments to address the concerns about novelty, evaluation coverage, and routing calibration.

1. Motivation and novelty. We agree that the distinction from existing methods should be clearer. A prompt memory may contain useful information, yet incomplete inputs can still retrieve inappropriate prompts. Existing memory prompting methods primarily learn retrieval through task supervision, while full-modality distillation generally transfers features or predictions. PPC instead uses complete training pairs to directly supervise which shared memory slots should be accessed under incomplete observations.

Two design choices are central: dense calibration provides corrective gradients even to slots excluded by the current sparse top-k retrieval, while our conditional-projection analysis characterizes what the incomplete-view router can learn without reconstructing unavailable information. The complete-view branch is removed at inference.

We have clarified this motivation in the Introduction. The information-matched controls in Table 2 support this distinction: PPC achieves 51.5 F1-Macro, compared with 49.3 for feature distillation, 47.8 for logit distillation, and 47.9 for the no-calibration control. This suggests that supervising memory access provides benefits beyond complete-view training information alone.

2. Missing-rate evaluation and routing analysis. We agree that evaluating only 70% missingness was insufficient. We have added Figure 3, covering 10%, 30%, 50%, 70%, and 90% image missingness on MM-IMDb, with separate training and matched-rate testing. PPC leads the compared methods at 50%, 70%, and 90%, while other methods perform better at lower missing rates.

Beyond benchmark performance, Tables 3–4 and Appendix G.2 examine routing behavior directly. Calibration increases missing-image Overlap@5 from 63.02% to 86.10% and reduces dense routing KL from 0.26 to 0.15. Under a fixed-predictor intervention, substituting the paired complete-view reference route improves missing-image F1-Macro from 49.65 to 51.11, whereas shuffled and random routes perform worse. These results connect routing alignment with its predictive utility, rather than relying solely on aggregate performance metrics.

3. Calibration-specific evaluation. We appreciate the suggestion to evaluate calibration more directly. PPC calibrates memory-routing distributions against complete-view references, rather than class prediction probabilities. Thus, classification ECE measures a different property and does not directly assess the intended routing alignment.

In addition to routing overlap, distributional discrepancy, and interventions, we have added Section 4.7 and Appendix G.4 to examine entropy-based confidence. Posterior entropy shows a weak negative correlation with routing utility (Spearman ρ=−0.09748\rho = -0.09748, 95% CI [−0.11798,−0.06309][-0.11798, -0.06309]). Paired routing reduces BCE by 0.002677 on average, and confidence predicts beneficial interventions with AUROC 0.5919 (95% CI [0.5721,0.6068][0.5721, 0.6068]). These results indicate a weak but measurable association, supporting entropy as a soft weighting cue rather than a guarantee of routing correctness. Its practical contribution is further examined by the confidence-weighting ablation in Table 2.

4. Equation (11). Thank you for identifying the potential division-by-zero issue. We now clarify that Ewarm=10>0E_{\mathrm{warm}} = 10 > 0 in our experiments. Disabling warm-up directly sets β(e)=βmax⁡\beta(e) = \beta_{\max} without division, resolving the ambiguity.

5. Clarity and presentation. We have simplified the abstract, clarified the motivation for supervising memory access, and reorganized Tables 2 and 4 with labeled experimental comparisons. Section 3 specifies the routing distributions and their slot-wise values. These targeted changes aim to improve readability without altering the methodology.

We hope these revisions and additional experiments address your main concerns and clarify PPC's contribution. We would be grateful if you could reconsider your initial assessment in light of the new evidence. Thank you again for your thoughtful feedback.

Thank you for your detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.

However, I still believe that the paper lacks substantial experimental validation that should have been included in the original submission. The newly added missing-rate evaluation is limited to image missingness on MM-IMDb, while evaluations for text-missing and both-missing scenarios remain absent. Moreover, t-SNE visualizations or other qualitative analyses are still missing. These are common and important evaluations for methods addressing missing modalities. The additional results also show that PPC does not consistently outperform existing methods at lower missing rates. These substantial experimental gaps make the paper feel more like a preliminary experimental draft than a mature, publication-ready submission.

Combined with my remaining concerns, I believe the paper is not yet ready for publication. Therefore, I am still inclined to recommend rejection.

Reviewer_LHeZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Privileged Posterior Calibration (PPC), which applies this supervision to distributions over shared prompt-memory slots. The framework transfers complete-view routing distributions to an observed-view router through a confidence-weighted dense objective, while retaining sparse retrieval for prediction. Specifically, the complete training counterpart supplies a soft routing target, and a dense, confidence-weighted objective teaches the observed router to approximate complete-view routing averaged over the missing evidence compatible with its input. Additionally, prototype initialization and label-neighborhood regularization give the routing target semantic structure. The authors evaluate the proposed method on three vision–language benchmarks. The experimental results demonstrate improved performance compared with existing methods.

Critical strengths
  1. How to retrieve useful prompts from memory under incomplete observations is a question worth investigating, and the controlled ablations and routing interventions provide evidence for the benefit of explicitly supervising this retrieval process.
  2. The paper is well organized, and the overall framework is easy to follow. The figures also help explain the proposed method.
Critical weaknesses
  1. PPC introduces privileged distillation into MemPrompt-style routing under a stronger complete-pair training assumption. Despite its methodological differences and empirical support, the technical advance beyond combining memory prompting with existing full-modality distillation remains unclear.
  2. PPC uses routing entropy to weight the supervision in Eq. (5). Although the ablation supports the benefit of this weighting, the paper provides no direct evidence that lower-entropy targets correspond to more reliable routing.
  3. While Appendix G.3 extends the routing-calibration ablation to other datasets, the effects of confidence weighting, prototype initialization, and semantic regularization are only examined on MM-IMDb under 70% image missingness. Their effectiveness across other datasets and missing-modality settings remains unclear.
Questions for authors
  1. In Eq. (5), the authors use routing entropy to estimate confidence. Could the authors provide evidence that lower entropy corresponds to more reliable prompt selection?
  2. Table 2 only reports component ablations on MM-IMDb under 70% image missingness. Could the authors provide additional ablation results on other datasets and missing-modality settings to verify the effectiveness of each component?
1 comment

We thank the reviewer for the thoughtful assessment and constructive feedback. We appreciate the recognition of the problem's importance and the value of our controlled ablations and routing interventions. We have revised the manuscript and added further analyses to clarify the technical contribution and address the concerns raised.

1. Technical contribution beyond privileged distillation. We agree that PPC builds on established ideas in privileged learning and memory-based prompting. Our main contribution lies in what is transferred and how this supervision affects memory retrieval. Rather than aligning complete-view features or prediction logits, PPC supervises the observed-view distribution over shared memory slots—the decision that determines which prompts enter the predictor.

This distinction is reflected in two design choices. First, dense calibration provides gradients to memory slots excluded by the student's current sparse top-k selection. Second, the conditional-projection analysis characterizes the routing target learnable from incomplete observations, without requiring the student to recover an unavailable complete view.

The information-matched controls in Table 2 examine whether this supervision offers benefits beyond access to complete training pairs. PPC achieves 51.5 F1-Macro, compared with 49.3 for complete-view feature distillation, 47.8 for logit distillation, and 47.9 for the architecture-matched no-calibration control. Moreover, Table 3 shows that calibration increases missing-image routing Overlap@5 from 63.02% to 86.10%, while Table 4 demonstrates that sample-matched complete-view routing improves prediction relative to observed and shuffled routing. Together, these results support the value of supervising memory access directly, rather than relying on complete-view supervision alone.

2. Reliability of entropy-based confidence. We agree that low entropy does not necessarily imply correct routing. To examine this assumption directly, we have added Section 4.7 and Appendix G.4, evaluating whether posterior entropy is associated with the predictive utility of retrieved prompts.

We keep the predictor and incomplete input fixed, replace only the visual-memory routing with its paired complete-view reference, and measure the resulting reduction in prediction BCE. Posterior entropy shows a weak but statistically detectable negative correlation with routing utility (Spearman ρ = −0.09748, 95% CI [−0.11798, −0.06309]). The paired routing intervention reduces BCE by 0.002677 on average (95% CI [+0.001372, +0.004155]), while entropy-derived confidence identifies beneficial interventions with AUROC 0.5919 (95% CI [0.5721, 0.6068]).

These results provide direct evidence of a modest association between routing concentration and predictive utility. We do not interpret entropy as a guarantee of correctness, but rather as a soft weighting signal. The confidence-weighting ablation in Table 2 provides complementary evidence of its practical contribution during training.

3. Effectiveness across datasets and missingness settings. We appreciate the reviewer's acknowledgment of the cross-task routing-calibration controls in Appendix G.3. These comparisons extend the evidence for the core calibration mechanism to Hateful Memes and Food101, where PPC exceeds the corresponding no-calibration controls by 0.98 AUROC and 1.16 accuracy points, respectively.

We have also added Figure 3, evaluating MM-IMDb at image-missing rates of 10%, 30%, 50%, 70%, and 90%. PPC performs best among the compared methods at 50%, 70%, and 90% missingness, indicating that its advantages are not confined to the original 70% setting.

We agree that a full cross-dataset ablation of every auxiliary component would provide stronger evidence. The current experiments establish their individual contributions under the controlled MM-IMDb setting, while the additional cross-task comparisons and missing-rate evaluations examine the broader effectiveness of the complete method and its central routing-calibration mechanism. We therefore distinguish evidence for the complete method from evidence for its individual components.

We hope these additional experiments and clarifications help address the reviewer's concerns. We appreciate the opportunity to strengthen both the explanation and empirical evaluation of PPC.

Reviewer_Dc4QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors situate this research in the space of memory-driven prompting to address the problem of an absent modality at deployment for multimodal predictors. The paper introduces Privileged Posterior Calibration (PPC) by formulating complete-view supervision of shared prompt memory as well as the conditional-projection target for the incomplete-view router. The supervision uses a confidence-weighted dense objective, while PPC retains sparse retrieval for prediction. PPC is evaluated on three vision-language benchmarks under the RAGPT/ANGA protocol, performing better than reproduced MemPrompt in all nine settings included.

Critical strengths
  1. The mathematical exposition in Section 3 is clear and elaborate, making the PPC formulation transparent. The training objective is also clearly defined. This is a considerable strength of the paper.
  2. The authors conduct extensive benchmarking (Table 1) to establish the efficacy of PPC. They consider not only other Memory-driven prompt methods such as MemPrompt, but also Modality-invariant learning methods, VLP-based methods, and Modality reconstruction methods.

Overall reasons to accept outweigh reasons to reject.

Critical weaknesses
  1. The main results in Table 1, especially for reproduced MemPrompt or PPC, are missing confidence intervals or standard deviations. Thus, we are unable to determine if the performance improvement is statistically significant. A lack of statistical significance would substantially weaken the conclusions of the paper.
Questions for authors
  1. Could the authors clarify the procedure to determine which vision-language benchmarks to include for the evaluation? Were any other benchmarks considered?
  2. Overall, could the authors explain the motivation for the specific formulation of PPC as presented? Were any alternate formulations or approaches considered? Was this approach adopted because of specific desirable properties?
2 comments

We thank the reviewer for the encouraging assessment and constructive suggestions. We are glad that the mathematical formulation and the breadth of our experimental comparisons were appreciated. We have revised the manuscript to address the concerns raised.

1. Statistical significance. We agree that the improvements should be supported by statistical evidence rather than single-run results alone. We have therefore added significance tests comparing PPC with our reproduced MemPrompt baseline across all nine settings in Table 1, using five independent training seeds per method. All nine comparisons yield p<10−3p < 10^{-3}, with relative improvements ranging from 2.07% to 9.77%. In addition, Table 2 reports mean ± standard deviation over three seeds for the component ablations and matched supervision controls, providing further evidence of the stability of the proposed method.

We have also extended our evaluation to image-missing rates of 10%, 30%, 50%, 70%, and 90% on MM-IMDb (Figure 3). PPC outperforms the compared methods at 50%, 70%, and 90% missingness, suggesting that its benefits are not restricted to the original 70% setting.

2. Benchmark selection. We selected Hateful Memes, MM-IMDb, and Food101 because they cover three different prediction tasks: binary classification, multi-label classification, and fine-grained multi-class classification. They also support direct comparisons with existing approaches under the established RAGPT/ANGA missing-modality protocol. We have clarified this motivation in Section 4.1.

3. Motivation for the PPC formulation. Our main consideration is that missing-modality inputs cannot recover the exact routing decisions made with complete information. Rather than distilling complete-view features or logits, PPC directly supervises the distribution over shared memory slots, which determines which prompts are retrieved. Dense calibration provides learning signals even for slots excluded by the student's current top-k selection, while sparse retrieval is retained for efficient inference. The confidence weighting further reduces the influence of diffuse posterior targets.

Section 3 explains these design choices and their conditional-projection interpretation, while the revised Introduction further clarifies the motivation. The alternative supervision experiments in Table 2 also provide empirical support: PPC achieves 51.5 F1-Macro, compared with 49.3 for complete-view feature distillation and 47.9 for the architecture-matched no-calibration control.

We hope these additional results and clarifications address the reviewer's concerns, and we appreciate the opportunity to improve the paper.

Reviewer_Dc4QOpenReview ↗

Response to Rebuttal

The reviewer appreciates the response and clarifications from the authors. The inclusion of the confidence intervals improves the credibility of the results. The reviewer would like to maintain the rating while emphasizing acceptance.

Comments on the paper

1 comment

We thank the reviewers for their continued feedback. Given the remaining concerns about experimental completeness, we would like to clarify why we believe the revised manuscript provides substantial evidence for PPC's central contribution.

1. Experimental coverage. Our evaluation is not limited to a single dataset or missingness scenario. Table 1 compares PPC with established methods across three datasets and all three missing patterns (text-missing, image-missing, and mixed-missing). PPC improves over reproduced MemPrompt in all nine settings, with five-seed significance tests supporting these improvements. We also provide cross-task routing-calibration controls on Hateful Memes and Food101 in Appendix G.3.

The newly added Figure 3 extends the evaluation across five missing rates on MM-IMDb. PPC performs best at 50%, 70%, and 90% image missingness, while we also report where other methods lead at lower rates. We do not claim that PPC is universally superior under every condition.

2. Validation of the core contribution. PPC's main contribution is the use of complete-view information to supervise memory-access decisions under incomplete observations. Accordingly, our experiments examine not only final prediction performance but also whether this supervision improves the intended routing mechanism.

Table 2 includes component ablations, feature/logit distillation baselines, and an architecture-matched no-calibration control. Tables 3–4 and Appendix G.2 further evaluate routing alignment, instance specificity, and the predictive value of routing decisions. The newly added Section 4.7 and Appendix G.4 examine the relationship between posterior confidence and routing utility.

Together, these experiments address three important questions: whether routing supervision provides benefits beyond privileged information alone, whether it improves memory access, and whether the resulting routing decisions matter for prediction. We believe this constitutes substantial validation of PPC's central technical claim.

3. Scope of the remaining concerns. We acknowledge that extending the missing-rate experiments to additional missing patterns and providing qualitative visualizations would further strengthen the paper. However, the existing main benchmarks already cover all three missing patterns, while the additional missing-rate experiments investigate robustness in a representative setting.

Similarly, t-SNE visualizations could provide complementary insights into learned representations, but PPC primarily concerns memory routing. Our routing-specific diagnostics and controlled interventions directly evaluate this mechanism.

We therefore respectfully believe that these remaining requests concern the breadth of additional analysis rather than a lack of experimental validation. While further experiments could certainly enrich the study, we do not believe their absence makes the current work a preliminary experimental draft.

We hope the Area Chairs and reviewers will consider the correspondence between PPC's technical contribution and the evidence already provided when evaluating the revised submission. We sincerely appreciate their time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.