Pointwise Multimodal Information Budget Regularization for Multimodal Learning

Multimodal learning systems integrate complementary information from different modalities and consequently demonstrate superior performance over unimodal methods. However, most existing multimodal fusion approaches assume that the modalities invariably provide synergistic benefits, overlooking the modality conflicts and joint failures that commonly arise in real-world deployment. This paper proposes a pointwise multimodal information budget framework that systematically distinguishes three regimes of multimodal interaction at the information-theoretic level: the synergistic regime, in which the modalities provide mutually consistent positive evidence; the conflicting regime, in which some modalities introduce contradictory information that interferes with the prediction; and the joint-failure regime, in which none of the modalities contains discriminative signals. The pointwise multimodal information budget adjusts the strength of the constraints according to the effective information contribution of each modality for the current sample. The multimodala dynamic routing mechanism uses the sign of pointwise mutual information as a supervisory signal to learn sample-wise modality reliability and enable branch selection. Extensive experiments on three multimodal benchmarks demonstrate the effectiveness and superiority of our proposed PMIB.