- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies memory routing for incomplete multimodal learning, aiming to address the challenge of modality missing at deployment. To this end, the paper proposes Privileged Posterior Calibration (PPC), which leverages complete image-text pairs available during training to supervise how inputs with missing modalities retrieve prompts from the prompt memory. The experimental results appear promising.
Critical strengths
-
The paper proposes a method for handling missing modalities at test time.
-
The paper provides theoretical analysis of the proposed method.
-
The experimental setup is relatively comprehensive, with sufficient baseline comparisons and competitive performance.
Critical weaknesses
-
Compared with existing methods, the motivation and novelty of the proposed approach are not sufficiently compelling. In addition, the paper is not clearly written.
-
The experimental evaluation is substantially insufficient. The paper only evaluates a 70% missing rate, rather than covering a broader range of missing rates (e.g., 10%–90%) as in prior work. Moreover, the evaluation relies solely on performance metrics and lacks complementary analyses such as visualizations.
Questions for authors
-
What are the key differences between the proposed method and existing approaches, and which limitations of prior methods does it aim to address? The paper lacks sufficient discussion of the limitations of existing methods.
-
Eq. 11 appears to have a potential division-by-zero issue.
-
Although the paper claims to perform “posterior calibration,” it does not report calibration-specific evaluations such as calibration error, ECE, or reliability analysis.
-
Prior work such as ANGA evaluates performance across missing rates ranging from 10% to 90%, whereas this paper only considers a 70% missing rate. Results under additional missing rates should be provided for a more comprehensive comparison. Moreover, beyond quantitative performance metrics, the paper would benefit from relevant visualization experiments to provide more complete empirical evidence.
-
The heavy use of terminology makes the paper difficult to follow. In the abstract, the descriptions of the motivation and method are overly compressed, whereas the discussion of experimental results is disproportionately detailed. In addition, the notation should more clearly distinguish scalars from vectors, and the layouts of Tables 2 and 4 should be improved for better readability.
2 comments
Thank you for your detailed response and additional experiments. I appreciate the authors' efforts to address my concerns.
However, I still believe that the paper lacks substantial experimental validation that should have been included in the original submission. The newly added missing-rate evaluation is limited to image missingness on MM-IMDb, while evaluations for text-missing and both-missing scenarios remain absent. Moreover, t-SNE visualizations or other qualitative analyses are still missing. These are common and important evaluations for methods addressing missing modalities. The additional results also show that PPC does not consistently outperform existing methods at lower missing rates. These substantial experimental gaps make the paper feel more like a preliminary experimental draft than a mature, publication-ready submission.
Combined with my remaining concerns, I believe the paper is not yet ready for publication. Therefore, I am still inclined to recommend rejection.
We thank the reviewer for the careful evaluation and constructive suggestions. We appreciate the recognition of our theoretical analysis and experimental comparisons. We have revised the manuscript and added experiments to address the concerns about novelty, evaluation coverage, and routing calibration.
1. Motivation and novelty. We agree that the distinction from existing methods should be clearer. A prompt memory may contain useful information, yet incomplete inputs can still retrieve inappropriate prompts. Existing memory prompting methods primarily learn retrieval through task supervision, while full-modality distillation generally transfers features or predictions. PPC instead uses complete training pairs to directly supervise which shared memory slots should be accessed under incomplete observations.
Two design choices are central: dense calibration provides corrective gradients even to slots excluded by the current sparse top-k retrieval, while our conditional-projection analysis characterizes what the incomplete-view router can learn without reconstructing unavailable information. The complete-view branch is removed at inference.
We have clarified this motivation in the Introduction. The information-matched controls in Table 2 support this distinction: PPC achieves 51.5 F1-Macro, compared with 49.3 for feature distillation, 47.8 for logit distillation, and 47.9 for the no-calibration control. This suggests that supervising memory access provides benefits beyond complete-view training information alone.
2. Missing-rate evaluation and routing analysis. We agree that evaluating only 70% missingness was insufficient. We have added Figure 3, covering 10%, 30%, 50%, 70%, and 90% image missingness on MM-IMDb, with separate training and matched-rate testing. PPC leads the compared methods at 50%, 70%, and 90%, while other methods perform better at lower missing rates.
Beyond benchmark performance, Tables 3–4 and Appendix G.2 examine routing behavior directly. Calibration increases missing-image Overlap@5 from 63.02% to 86.10% and reduces dense routing KL from 0.26 to 0.15. Under a fixed-predictor intervention, substituting the paired complete-view reference route improves missing-image F1-Macro from 49.65 to 51.11, whereas shuffled and random routes perform worse. These results connect routing alignment with its predictive utility, rather than relying solely on aggregate performance metrics.
3. Calibration-specific evaluation. We appreciate the suggestion to evaluate calibration more directly. PPC calibrates memory-routing distributions against complete-view references, rather than class prediction probabilities. Thus, classification ECE measures a different property and does not directly assess the intended routing alignment.
In addition to routing overlap, distributional discrepancy, and interventions, we have added Section 4.7 and Appendix G.4 to examine entropy-based confidence. Posterior entropy shows a weak negative correlation with routing utility (Spearman ρ=−0.09748, 95% CI [−0.11798,−0.06309]). Paired routing reduces BCE by 0.002677 on average, and confidence predicts beneficial interventions with AUROC 0.5919 (95% CI [0.5721,0.6068]). These results indicate a weak but measurable association, supporting entropy as a soft weighting cue rather than a guarantee of routing correctness. Its practical contribution is further examined by the confidence-weighting ablation in Table 2.
4. Equation (11). Thank you for identifying the potential division-by-zero issue. We now clarify that Ewarm=10>0 in our experiments. Disabling warm-up directly sets β(e)=βmax without division, resolving the ambiguity.
5. Clarity and presentation. We have simplified the abstract, clarified the motivation for supervising memory access, and reorganized Tables 2 and 4 with labeled experimental comparisons. Section 3 specifies the routing distributions and their slot-wise values. These targeted changes aim to improve readability without altering the methodology.
We hope these revisions and additional experiments address your main concerns and clarify PPC's contribution. We would be grateful if you could reconsider your initial assessment in light of the new evidence. Thank you again for your thoughtful feedback.