acceptodds
Under review as a conference paper at ICLR 2027

Pointwise Multimodal Information Budget Regularization for Multimodal Learning

Abstract

Multimodal learning systems integrate complementary information from different modalities and consequently demonstrate superior performance over unimodal methods. However, most existing multimodal fusion approaches assume that the modalities invariably provide synergistic benefits, overlooking the modality conflicts and joint failures that commonly arise in real-world deployment. This paper proposes a pointwise multimodal information budget framework that systematically distinguishes three regimes of multimodal interaction at the information-theoretic level: the synergistic regime, in which the modalities provide mutually consistent positive evidence; the conflicting regime, in which some modalities introduce contradictory information that interferes with the prediction; and the joint-failure regime, in which none of the modalities contains discriminative signals. The pointwise multimodal information budget adjusts the strength of the constraints according to the effective information contribution of each modality for the current sample. The multimodala dynamic routing mechanism uses the sign of pointwise mutual information as a supervisory signal to learn sample-wise modality reliability and enable branch selection. Extensive experiments on three multimodal benchmarks demonstrate the effectiveness and superiority of our proposed PMIB.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Sentiment Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WHmcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a pointwise multimodal information estimation framework based on three common multimodal interaction patterns—namely, synergy, conflict, and joint failure—and estimates pointwise mutual information through unimodal predictions and class priors. Extensive experiments on three multimodal benchmark datasets demonstrate the effectiveness of the proposed PMIB method.

Critical strengths

1、This paper proposes a fusion framework based on pointwise mutual information estimation, which can achieve information accumulation under synergy, reliably select branches in conflict situations, and effectively suppress errors in cases of joint failure.

2、It demonstrates promising performance on multimodal sentiment analysis tasks.

Critical weaknesses

1、For the three-modality interaction patterns in Figure 1 (c), further clarification is needed regarding how the proportions of synergy, conflict, and failure were obtained for each dataset. If these were compiled by the authors themselves, how were they measured?

2、The authors use unimodal predictions and class priors to estimate pointwise mutual information. If the unimodal predictor produces incorrect predictions, is it still valid to estimate pointwise mutual information based on the probabilities predicted from these erroneous unimodal predictions? Furthermore, the authors should provide experimental results using only a single modality to support estimating pointwise mutual information through unimodal predictions and class priors.

3、The authors primarily conducted experiments on multimodal sentiment analysis tasks. For instance, when applying the proposed method to MHD (multimodal humor detection, UR-FUNNY dataset) and MSD (multimodal sarcasm detection, MUStARD dataset), it is necessary to exploit the "conflict" information between different modalities for discrimination. Is the proposed method still effective under such circumstances?

4、Recent papers such as "Learning Optimal Multimodal Information Bottleneck Representations" and "Efficient Quantification of Multimodal Interaction at Sample Level" also use mutual information as a basis to address the challenges of synergy, conflict, and redundancy in multimodal learning. Since this paper likewise employs mutual information to tackle the same problems, what advantages does it offer?

5、The code is not available, resulting in poor reproducibility.

6、Is "multimodala" in the abstract a typographical error?

7、Should the full name of the MSA task be provided where it first appears in line 73?

Questions for authors

The authors need to address all the issues raised in the Weaknesses.

Reviewer_u3nEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper uses unimodal scores to group samples into agreement, conflict, and joint-failure cases. It then adjusts modality weights and fusion confidence for each sample, aiming to preserve useful information, reduce interference from unreliable modalities, and avoid overconfidence when reliable evidence is lacking.

Critical strengths

Analyzing multimodal learning from an information-theoretic perspective is a promising starting point, with considerable room for developing methods in this direction. The paper provides fairly comprehensive experimental analyses and ablation studies. Releasing the source code would further strengthen confidence in the results.

Critical weaknesses

My main concern is the link between unimodal score signs and the usefulness of joint information. Eq.4, 8, and 11 treat two non-positive scores as joint failure and limit fusion confidence. But consider the XOR task:

X1,X2∼i.i.d.Bernoulli(1/2),Y=X1⊕X2.X_1,X_2\overset{\mathrm{i.i.d.}}{\sim}\mathrm{Bernoulli}(1/2),\qquad Y=X_1\oplus X_2.

With true conditional probabilities, P(y∣xk)=P(y)=1/2P(y\mid x _k)=P(y)=1/2, so v1=v2=0v _1=v _2=0 and Bα=0B _\alpha=0. Yet the joint input fully determines the label:

I(Y;X1)=I(Y;X2)=0,I(Y;X1,X2)=ln⁡2.I(Y;X _1)=I(Y;X _2)=0,\qquad I(Y;X _1,X _2)=\ln 2.

This leaves a gap in the justification for the budget: weak unimodal evidence does not imply weak joint information. The zero-loss proofs do not resolve this issue. Controlled experiments would help establish when this rule is appropriate.

Also, vk>0v _k>0 in Eq.2 only means pk(y∣xk)>π(y)p _k(y\mid x _k)>\pi(y). With ten equally likely classes, a model assigning 0.120.12 to the true class and 0.60.6 to a wrong class still satisfies this condition. Thus, positive scores do not establish correct predictions or agreement between modalities. Please report accuracy among samples with vk>0v _k>0 and clarify how π(y)\pi(y) is estimated.

Eq.7 should also account for the prediction errors acknowledged in Eq.23. For model probabilities, the exact relationship is

EP[vf−vk]=IP(Y;Xkˉ∣Xk)+Dk−Df,\mathbb{E} _P[v _f-v _k]=I _P(Y;X _{\bar{k}}\mid X _k)+D _k-D _f,

where DkD_k and DfD_f are the expected KL divergences from the true conditional distributions to the corresponding model predictions. The expectation can therefore be negative if the fusion model’s error is sufficiently large; conditional MI non-negativity alone does not guarantee otherwise.

And there is a minor notation issue, S\mathcal{S} denotes the agreement set, while SS denotes synergistic information. These similar symbols make the discussion harder to follow.

Questions for authors

Can the proposed budget handle cases where individual modalities contain no label information, but the joint input fully determines the label, such as the XOR example discussed in the weaknesses? This is my main concern about the paper. Addressing it convincingly would make me more willing to raise my rating.

Definition 1 specifies S\mathcal{S}, C\mathcal{C}, and F\mathcal{F} only for two modalities, while all three datasets used in the experiments have three modalities. Please clarify how these sets are defined for more than two modalities.

How is B~α\widetilde{B} _\alpha defined in Eq.10 and Eq.13? The earlier text defines BαB _\alpha but does not explain the relationship between B~α\widetilde{B} _\alpha and BαB _\alpha.

Eq.17 includes a regression loss, but it does not explain how the regression predictions are obtained. Are they produced by a regression head separate from the classification head?

I would also ask the authors to provide the source code needed to reproduce the results, as this would substantially strengthen confidence in the paper’s findings.

Reviewer_nahDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses a real limitation of multimodal fusion: most methods implicitly assume that modalities always provide mutually supportive, task-relevant evidence, whereas in practice a given sample can fall into one of three regimes — (i) synergy, where both modalities lift the true class above its prior; (ii) conflict, where one modality is informative and the other misinformative; or (iii) joint failure, where no modality rises above the prior. The authors introduce a pointwise quantity v_k = log(p_k(y|x_k)/π(y)), a log-ratio of the unimodal prediction to the class prior, and use its sign to assign each sample to one of the three regimes. The per-modality evidence is then collapsed into a single scalar "information budget" B_α = max_k v_k + α[min_k v_k]_+, on which two-sided hinge constraints are imposed (a lower bound enforcing that supported information be realized and an upper bound capping unsupported confidence). A routing module g_k = σ(φ_k(E_k)) is supervised by the PMI sign (via binary cross-entropy) to down-weight negative-information modalities at inference, while α is estimated online from batch co-information to adapt to dataset-specific redundancy.

Critical strengths

S1. A principled, pointwise (sample-level) operationalization of interaction regimes. The core insight — that the distribution-level non-negativity of conditional mutual information, E[v_f − v_k] = I(Y; X_ķ | X_k) ≥ 0, does not hold pointwise, and that the pointwise violations (v_f − v_k < 0) concentrate precisely on failure samples. Encoding the three regimes purely through the sign pattern of v_k, and collapsing them into a single scalar budget B_α (Eqs. 8–9) that automatically realizes synergy amplification, conflict routing, and failure suppression without explicit regime dispatch, is elegant.

S2. The constraints carry concrete, verifiable semantics rather than being ad-hoc regularizers. Propositions 1–3 map the two hinge bounds onto known quantities: the lower bound approaches the product-of-experts form under synergy (p_f(y|x) ≥ e^m·p_1p_2/π(y)) and equals pointwise CMI non-negativity under conflict (v_f − v_r ≥ m); the upper bound yields a provable rejection guarantee under failure (max_c p_f(c|x) ≤ e^ε π_max < 1, zero false-positive rate).

S3. Experiments on MOSI, MOSEI, and SIMSv2 report gains over recent fusion/representation-learning baselines, plus ablations and qualitative analyses.

Critical weaknesses

W1. The gains are confounded by the text backbone, with no controlled comparison isolating the method. PMIB uses DeBERTa-V3 text features (App. C.2), while most baselines rely on weaker text encoders (BERT/GloVe-era); the authors neither re-implement baselines under the same DeBERTa-V3 backbone, nor ablate the backbone itself. Since the text backbone largely dominates performance in MSA, the claimed superiority cannot be attributed to the information-budget regularization and likely reflects the stronger language model alone.

W2. The "significantly outperforms" claim is unsupported, and the novelty appears incremental. (a) PMIB is not uniformly best — it trails EBMC by 2.67 points on MOSEI Acc-7 and falls behind EBMC/TSD/FUSE-Net/EMOE on MOSEI MAE; (b)The relation to reliability-aware fusion (Han et al. ,2022; Zhang et al. ,2023b) and to the synergy/redundancy decomposition of Liang et al. (2023) is not clearly delineated.

W3. The writing is sloppy, with multiple typos and punctuation errors, e.g., "multimodala", "…interaction information Under…" (missing period and capitalization), and "Equation equation ".

Questions for authors

The relevant concerns are presented in the Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.