acceptodds
Under review as a conference paper at ICLR 2027

Beyond PID Synergy: Fusion Value and Pairing Value for Multimodal Bayes Risk

Abstract

Multimodal fusion is commonly motivated by partial information decomposition (PID) synergy, and a growing diagnostic literature treats this atom as the reason to combine two views. In this work, we show that this identification does not hold at Bayes optimality. The logarithmic loss gap between the better unimodal Bayes predictor and the joint Bayes predictor is not synergy alone, since it also carries the weaker unique atom. We call this gap fusion value. A second gap separates early fusion from the stacking of two calibrated unimodal posteriors, and this gap is not a linear function of any Williams and Beer quadruple. We call it pairing value. Pairing value lies between zero and fusion value, with AND attaining the lower bound under a strictly positive synergy atom and XOR attaining the upper bound with one bit of both. Adding a named view and deleting it produce the same Bayes gap, so the empirical missing modality drop reads the same quantity under two names. For trained models, the observed fusion gain decomposes into fusion value plus a signed excess risk remainder. An operational protocol built from held out logarithmic loss keeps this remainder near zero whenever a single view already saturates the label. On language saturated sentiment, the remainder is near zero, because the better singleton already captures the label. On caption matching, both views are necessary and late fusion stays at chance, since the label lives only in the pair. The two indices therefore separate the decision to retain a weak view from the decision to interact early.

Primary area:
learning theory
Keywords:
Partial information decomposition, Multimodal fusion, Bayes risk

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iMk2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes fusion value, the benefit of combining two views over the better individual view, from pairing value, the benefit of early fusion over stacking unimodal posteriors. It connects these quantities to Bayes risk and PID, and studies their behavior through synthetic examples and multimodal experiments.

Critical strengths

The distinction between retaining a second modality and enabling early interaction is conceptually useful. The AND/XOR examples clearly demonstrate why PID synergy alone cannot determine the need for early fusion. The discussion of excess risk also identifies an important gap between information-theoretic quantities and measured model performance.

Critical weaknesses

The linear error-improvement bound stated after Corollary 4 is incorrect. For a uniform binary label, an uninformative first view, and a second view generated through a binary symmetric channel with crossover probability 0.4, the Bayes error improvement is 0.1, while the claimed bound is approximately 0.0145. The empirical interpretation also needs stronger justification. Differences between fitted models contain unknown approximation and optimization errors; calibration and small observed fusion gains do not establish that these errors are negligible. Consequently, the architectural conclusions on real datasets are less secure than the exact Bayes identities.

Questions for authors

How should the bound following Corollary 4 be corrected, and which downstream claims require revision? How can the approximation and optimization errors of the unimodal, stacking, and joint predictors be assessed when estimating FVI/PVI?

Reviewer_KomhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Starting from Bayes-optimal prediction, the paper uses FVI and PVI to distinguish the value of retaining a second modality from that of introducing early interaction. Examples such as AND and XOR illustrate the difference between PVI and PID synergy.

Critical strengths

Distinguishing whether a second modality is useful from whether early interaction is necessary is an insightful perspective. The comparison between AND and XOR is intuitive. Lemmas 2 and 4 also separate theoretical gains from model training errors, which helps interpret observed fusion gains.

Critical weaknesses

(1) The connection between the theoretical indices and actual model performance remains insufficiently established. For positive-class probabilities in binary classification, calibration does not guarantee the condition required by Lemma 3:

\;\not\Rightarrow\; \exists t_i:\ g_i(X_i)=t_i(\eta_i(X_i)).$$ Moreover, Eq.4 gives $$G=\text{FVI}+\varepsilon_{\mathrm{uni}}-\varepsilon_{\mathrm{joint}}, \qquad G\approx0\;\not\Rightarrow\;\text{FVI}\approx0,$$ where $\varepsilon_{\mathrm{uni}},\varepsilon_{\mathrm{joint}}\geq0$ are the corresponding excess risks. The authors therefore need to explain how these errors are controlled before using the operational indices to guide architecture selection. (2) Some mathematical statements in the main text need correction. The conclusion $\text{PVI}=0$ in Line 192 does not follow directly from the preceding equality: substituting $I(Y;\eta_1,\eta_2)=I(Y;X_{\mathrm{strong}})$ into the definition gives $\text{PVI}=\text{FVI}$, so $\text{FVI}=0$ would also need to be established. The relationship between classification error reduction and conditional mutual information in Lines 201-203 also needs clarification. (3) Most theoretical results follow directly from existing information-theoretic formulas, so the theoretical novelty is limited. For example, Theorem 1 follows directly from the relationship between Bayes logarithmic loss and conditional entropy, while FVI is identical to the existing MMI synergy: $$\begin{aligned} R^*_{\log}(X_i)-R^*_{\log}(X_1,X_2) &=H(Y\mid X_i)-H(Y\mid X_1,X_2)\\ &=I(Y;X_{3-i}\mid X_i),\\ S_{\mathrm{MMI}} &=I(Y;X_1,X_2)-\max_i I(Y;X_i)=\text{FVI}. \end{aligned}$$ The main novelty lies in PVI and its interpretation, but its added value for practical model selection has not been sufficiently demonstrated.
Questions for authors
  1. Why do the calibrated model outputs satisfy the condition in Lemma 3? What additional assumptions or error controls are needed to relate the operational pairing value to the theoretical PVI?
  2. Could the positive pairing values for copy and unique in Table 13 be due to inadequate training of the late fusion model? Please provide results with more thorough optimization and across random seeds, and explain how Sec.5.4 rules out underfitting of the joint model.
  3. Please clarify whether additional conditions are missing from Line 192 and Lines 201-203, and which conclusions remain valid after correction.
Reviewer_xwVDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

To address the limitation in multimodal learning that equates multimodal fusion gain with partial information decomposition (PID), this paper formalizes multimodal design via a Bayes risk perspective under logarithmic loss. The framework comprises three core components: i) Fusion Value characterizes the log-loss gap between the better unimodal predictor and the joint Bayes predictor, building the Bayes duality between adding and deleting a modality. ii) Pairing Value measures the gap between early fusion and stacking calibrated unimodal posteriors, proven non-linear in any Williams & Beer PID quadruple. iii) An operational protocol bridges theoretical Bayes to empirical models by decomposing finite-sample excess risks on calibrated log-loss.

Critical strengths
  1. The motivation is somewhat interesting for exploring the distinction between synergy and the weaker modality's unique information.
  2. The comparison between early and late fusion is illustrated by showing their separation via simple discrete cases.
  3. Some ablation experiments are conducted to support the claims.
Critical weaknesses
  1. The theoretical novelty is limited, as the core proofs rely primarily on elementary algebraic rearrangements and known textbook identities. For example, Theorem 1 solely substitutes the Williams & Beer decomposition into classical conditional entropy identities.
  2. This work claims that PID synergy is the dominant consensus in multimodal fusion, which misaligns with the truth.
  3. The empirical validations heavily depend on synthetic toy datasets such as sliced MNIST, which are very weak and not convincing enough.
  4. In Figure 4, the evaluation reports across all scales are approximately 50%, indicating that none of the proposed components are useful.
  5. The authors are encouraged to provide the real benchmark dataset, sample size, class distribution, training hyperparameters, and training trajectories. Furthermore, why does dropping either modality still yield the same test error of 0.525 across all scales?
  6. The paper focuses on multimodal fusion. However, multiple positions in the manuscript discuss the multi-view. The authors should carefully proofread and unify the entire article.
Questions for authors

Please see the critical weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.