acceptodds
Under review as a conference paper at ICLR 2027

UNCERTAINTY-GATED KNOWLEDGE ALIGNMENT FOR ROBUST MULTIMODAL LEARNING UNDER MISSING MODALITIES

Abstract

Multimodal models degrade sharply when entire modalities are missing at test time. Existing remedies either reconstruct the missing stream, learn modality‑agnostic features, or supplement observations with external knowledge; however, knowledge is typically injected with fixed, availability‑count heuristics (e.g., w_K = 1/(1+|A|)) that ignore how unreliable the observed evidence actually is, and information‑bottleneck variants are often stated without a tractable derivation. We propose UGKA (Uncertainty‑Gated Knowledge Alignment), a framework that couples three components: (i) a modality‑symmetric variational information bottleneck (MVIB) with a closed‑form Gaussian ELBO that yields modality‑invariant latents; (ii) uncertainty‑weighted fusion (UWF), a precision‑weighted product‑of‑experts that provably minimizes the expected NLL among linear fusion rules under calibrated heteroscedastic heads; and (iii) an uncertainty‑driven progressive retrieval (UPR) expert that conditions the knowledge readout’s fusion precision on the predictive entropy of the fused observation, so knowledge competes with sensors exactly when the observed evidence is unreliable. We provide complete derivations and proofs, and we validate UGKA in a fully reproducible protocol on real sensor data (UCI HAR with three heterogeneous modalities) and on a controlled knowledge‑grounded benchmark (KGM3) with per‑sample reliability heterogeneity, reporting mean±std over three seeds, missing‑rate sweeps, per‑modality ablations, and confusion analyses. Our headline finding is quantitative: when retrieved knowledge is conditionally informative, fusing it as a full precision‑weighted expert cuts the full‑range degradation on KGM3 from 45.0 to 24.1 points (accuracy at r=0.8: 54.3 vs. 75.2), while on real sensor data— where no external knowledge source exists—modality‑dropout training remains the dominant factor, and our learned availability‑free weighting performs on par with, but not better than, the availability‑count heuristic. We report these positive and negative results side by side; every number is produced by the released code.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Uncertainty‑Gated Knowledge Alignment for Robust Multimodal Learning Under Missing Modalities

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FFnKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes UGKA for multimodal learning with missing modalities. It uses an information bottleneck, uncertainty-based fusion, and knowledge retrieval. The main idea is to decide how much to rely on knowledge based on the reliability of the available modality.

The paper is overall not reaching the bar of ICLR venue, and I highly recommend the author to revise further and provide more details in motivation, experiment, proposed method and evaluation. But still, I am happy to give comments

Critical strengths

The motivation makes sense: having more available modalities does not necessarily mean having more reliable information. But the motivation is not a new thing in this domain. Other than that, no specific strength can be found in this paper.

Critical weaknesses

The proposed method does not show a clear advantage. The full model performs worse than simple availability-based weighting on KGM3 and ModDrop on HAR, making the benefit of its added complexity unclear. This is a critical weakness to reject the paper.

Some claims contradict the results. Figure 2’s caption says uncertainty gating performs better at high missing rates, while the reported numbers show otherwise. These inconsistencies make the conclusions harder to trust.

The theory does not fully support the claims. A lower bound on error becoming smaller does not show that the model actually makes fewer mistakes. The reliability of the learned uncertainty estimates is also not demonstrated.

The evaluation is limited. The experiments cover only one real dataset and one synthetic dataset, with mostly basic baselines. Stronger comparisons and a real external-knowledge setting are needed to establish the method’s value. This is a critical weakness to reject the paper.

Questions for authors

The knowledge source needs clarification. It is unclear how sample-specific knowledge is obtained during testing, particularly when all modalities are missing. The gains may mainly come from an extra informative input.

Can author clarify a little more on this point?

Reviewer_NXh9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper concerns the integration of external knowledge when certain modalities go missing at test-time. The authors propose an uncertainty-aware external knowledge integration method for dealing with such scenarios.

Critical strengths
  1. The experimental reporting is transparent. The authors report both positive and negative findings. Particularly, availability-count knowledge weighting surpassing the proposed full UGKA in most cases is an important negative result.

  2. The experiments with the synthetic KGM3 dataset provide a controlled study of the applicability of external knowledge in the missing modality scenario.

Critical weaknesses
  1. The uncertainty signal used to condition retrieval and knowledge weighting is the entropy of the softmax distribution over fused logits. This measures the dispersion of predicted class probabilities, but does not establish that epistemic uncertainty is adequately captured [a]. It is well known that neural networks can produce overconfident, miscalibrated predictions [b], and uncertainty estimates can deteriorate under distribution shift [c]. Although UGKA additionally learns heteroscedastic variance heads, the paper provides no calibration analysis demonstrating that these variances or the entropy signal are informative about prediction errors under missing or corrupted modalities. This is particularly consequential because the fusion optimality guarantees assume calibrated variances.

  2. The proposed method does not distinguish between aleatoric and epistemic uncertainty. In order to take into account the uncertainty inherent in the external knowledge source, it is important to explicitly account for aleatoric uncertainty.

  3. The ModDrop reference contains errors. The name of the co-author is Florian Nebout, not Francois Nebout. Also, the T-PAMI citation is 1692-1706, not 169–181.

  4. For UCI HAR, ModDrop is the strongest performer across all modality missing rates. For KGM3, UGKA with availability-count knowledge weighting is the strongest performer across all rates. Further in Table 2, UGKA with availability-count knowledge weighting significantly surpasses full UGKA. Thus, there is no demonstrated scenario where full UGKA is useful or needed. As the authors themselves acknowledge in the introduction, "existing knowledge-augmented fusion schemes" basically perform availability-count weighting, given these results, it is unclear why the proposed method is necessary in the first place.

References:

[a] Kendall and Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? NeurIPS 2017.
[b] Guo et al. . On Calibration of Modern Neural Networks. ICML 2017.
[c] Ovadia et al. Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. NeurIPS 2019.

Questions for authors

Refer to the weaknesses section.

Reviewer_zX9xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UGKA, a framework for robust multimodal learning under missing modalities. It combines a modality-symmetric variational information bottleneck (MVIB), uncertainty-weighted fusion (UWF), and uncertainty-driven progressive retrieval (UPR). The method uses predictive uncertainty to regulate knowledge retrieval and fusion. Experiments on UCI HAR and a synthetic benchmark (KGM3) examine robustness under different modality-missing rates.

Critical strengths
  1. Principled uncertainty-aware fusion framework. The paper presents a unified framework combining variational representations, precision-weighted fusion, and uncertainty-conditioned knowledge retrieval. The theoretical analysis provides useful motivation for the design under specific assumptions.

  2. Transparent experimental reporting. The authors report both positive and negative results, including cases where the proposed method fails to outperform simpler baselines. The reproducible experimental protocol and missing-rate evaluations are appreciated.

Critical weaknesses
  1. The core uncertainty-gating mechanism does not demonstrate empirical superiority. The main contribution is replacing availability-based knowledge weighting with uncertainty-conditioned precision. However, UGKA consistently underperforms the availability-weighted variant on KGM3 (72.6% vs. 75.2% at an 80% missing rate) and provides no improvement on UCI HAR. These results undermine the central motivation of the proposed gating mechanism.

  2. The proposed framework underperforms a simple baseline on real-world data. On UCI HAR, ModDrop consistently outperforms UGKA across all reported missing rates. For example, at an 80% missing rate, ModDrop achieves 48.5% accuracy compared with UGKA's 45.1%. Thus, the practical benefits of the additional architectural complexity are not demonstrated on real sensor data.

  3. Limited experimental validation and theoretical justification. The evaluation is restricted to one relatively small real-world dataset and one synthetic benchmark. Furthermore, the optimality of precision-weighted fusion relies on calibrated, conditionally independent estimation errors, but the paper does not empirically establish that these assumptions hold. The theoretical guarantees therefore provide limited evidence for the effectiveness of the implemented system.

Questions for authors

Q1. Why does uncertainty-conditioned knowledge weighting underperform the simpler availability-based heuristic on KGM3, despite the benchmark being designed with heterogeneous observation reliability?

Q2. Can the authors demonstrate improvements over ModDrop on additional real-world multimodal datasets?

Q3. Have the authors evaluated the calibration and error dependence of the modality-specific uncertainty estimates used in the theoretical analysis?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.