- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a way to separate the quantification of mechanistic and source redundancy. However, it has some serious flaws.
Critical strengths
The authors address an unsolved issue, but the solution proposed is not sound.
Critical weaknesses
The proposed method is not really able to quantify mechanistic and source redundancy separately, and on top has issues with the definition of unique information and the interpretation of local values.
Questions for authors
1- line : mechanistic redundancy appears in the AND gate, not the XOR gate.
2-The redundancy measure of Griffith 2014 is known to have substantial problems, being too demanding in terms of what common information is. For example, it vanishes when the joint distribution has full support, hence for the majority of distributions. This also render sit brittle to noise. An additional problem is that the approximation of GK used in this paper may actually help to address this fragility of the GK, but conceptually it is not desirable that the method works precisely when it does not approximate GK too faithfully. The authors do comment on the fragility of the GK measure, but not on this consequence for the role of its approximation.
3-Related to the above, if the proposed approach is to work when departing to some degree from a strict GK estimation, then the exact valid range of $\lambda $ is unknown.
4- There is an interpretation problem of redundancy at the sample level: consider definition of Eq 3 and the case of , and , for some . , and . Therefore, there is an interval in which their signs differ. It is hard to interpret the consequence of this as a sample-PID with no redundancy in that interval, simply because what you do have is that across the whole distribution of and lead to a consistent narrowing of the variance. That narrowing is a global property that happens across all . The sample-PID semantics are hard to interpret locally in the interval where zero values are obtained, but they do lead to a consistent redundancy value when averaged across . So I think there is a problem in the interpretability of sample-level PIDs.
5- In Eq 5 and 6 pick or , but be consistent.
6-In Eq 5 and 6 the meaning of and has not been described. The support of is not given. You need an explanation of and what this means in terms of samples.
7-In Eq 7, even if is close to , that does not enforce to be close to . A clear example is to define with values and , . For limit to 0, and are always infinitesimally close, and yet and . So similarity in the values of does not imply alignment of the distributions . This is different than the single Griffith et al approach, in that case really there is a single variable accounting for a single common .
8-The role of the shared prediction head is crucial to attenuate problems of the sort of point 7. However, this is not at all described when introduced. Even more, it is misleading that the use of subscripts is not consistent. the encoder should be source specific, , while is not source specific, isn't it? Only the common prediction head can constraint the misalignment of the type of point 7.
9-In fact, it is unclear from the main paper what is the role of the common prediction head. If are trained previously to using the common prediction head, that does not attenuate the potential misalignment of the two representations and the fact of having substantially different even with very similar .
10-The mapping from Eq 6 to Eq 7 is very nontrivial, the optimized objective function changes. The first term in Eq 7 it is misleading that it corresponds to , it is only monotonically paired. The second term uses Renyi entropy.
11-Not only the mapping from Eq 6 to 7 is nontrivial in terms of the mathematical derivation, it fundamentally alters the objective of the optimization. This is not well stated. When is noisy and within-class structure exists, minimizers of Eq 7 do not necessarily lead to good maximizers of Eq 6.
12-Related to the above, In Eq 6 the contribution of classes and their matched or unmatched predictions is weighted by their occurrence. In Eq 7 the normalization with | and reweights these contributions in a way that is not sufficiently noticed in terms of its implications.
13-There are multiple definitions of redundancy that appear inconsistent, and this is not clarified. Eq 3 and 41 follow the sign criterion, Eq 8 does not. Is it done on purpose that the sign criterion is used for total redundancy and not for source redundancy?
14-The necessity of the reassignment in line 226 is mathematically convenient, but conceptually indicates a substantial flaw. The problem is that data processing inequality does not hold pointwise, and hence even if is derived from , the info can be higher. This complicates even more the semantics of pointwise PIDs.
15- There is some conceptual problem between the fact that nonnegativity of the mechanistic information is enforced in line 226 and the fact that by doing so is updated to a value that can be higher, leading to smaller unique information. In fact you can even have , so that you create a negative unique. This asymmetry between enforcing nonnegativity of the mechanistic redundancy and creating negativity of the unique is hard to swallow. If the negative unique is to be taken as unique misinformation, why not having redundant mechanistic misinformation?
16-The framework inherits a problem from the Williams & Beer formulation that additionally affects the interpretability of mechanistic redundancy. This is the lack of fulfillment of the identity property. Consider , , and , where all three variables , , and have 1 bit. It is well known that quantifies 'same amount of information' as redundancy, even if the information is about a different variable and the variables are independent. This affects the separation of mechanistic and source redundancy. With the strict GK approach of Griffith et al 2014, in this case can be retrieved functionally from both sources, and hence . But the total redundancy is , because of the 'same amount of info issue'. This forces , despite the fact that there is no common mechanism beyond the common . When using the procedure of Eq 7 to relax GK, may capture more than one bit, up to two bits. In that case, may be zero, but the quantification of the total redundancy is still inadequate, if it is accepted that redundancy should not correspond to simply 'same amount of info'.
12 comments
Reviewer ye8L (Q3): "Related to the above, if the proposed approach is to work when departing to some degree from a strict GK estimation, then the exact valid range of lambda is unknown."
Choice of (Q3). We agree that no exact theoretical range for can be given, since the relative scale of the loss terms depends on the data. However, the valid range is bounded by two identifiable failure modes, corresponding to the two extremes of the relaxation. As , there is no constraint, and private information leaks into . As , SupGK approaches the strict quantity and inherits its fragility: on noisy, high-dimensional data, shared information is suppressed (as observed on AV-MNIST, App. M.1). We tested this on three examples with known :
| True | |||||||
|---|---|---|---|---|---|---|---|
| Two-bit copy (no shared info) | 0.000 | 0.201 | 0.280 | 0.056 | 0.000 | 0.000 | 0.000 |
| Partial source (QjeU) | 0.693 | 0.687 | 0.683 | 0.691 | 0.691 | 0.691 | 0.691 |
| Q16 example | 0.693 | 0.732 | 0.790 | 0.691 | 0.691 | 0.691 | 0.691 |
Table R1. Estimated (nats, mean over 3 seeds; std ≤ 0.001 for ).
Importantly, the right value can be identified without access to the true decomposition. Leakage has a visible symptom: when private information enters , the source-redundancy estimates of the two modalities disagree. Our criterion therefore selects the smallest at which cross-modal agreement is reached (App. M.1); choosing the smallest such value avoids the over-alignment regime, in which further alignment only suppresses shared information. In the two-bit copy and in the mixture without shared bit, the cross-modal MAE is still 0.07–0.21 nats at and drops to 0.000 at (3 seeds), so the criterion selects , where . When shared information is present (Q16 example), agreement is already reached at (MAE 0.000), and the criterion selects , as it does on AV-MNIST. On the Huntington's disease data, it selects , and source redundancy changes only from 29.4% to 29.5% between and 25. We have added this discussion to App. M.1 (see also App. O, P).
Response: We thank the reviewer for this example, which we verified: for , , so the local redundancy of Eq. 3 is zero. This zero has a clear meaning. Both sources narrow the distribution, but by different amounts: in this interval, already lies in the tail of the tighter while still in the bulk of the wider . For this particular outcome, the two sources disagree on whether is plausible, and this is what a zero local redundancy reports. Both sources narrow the distribution, but by different amounts: in this interval, already lies in the tail of the tighter while still in the bulk of the wider . For this particular outcome, the two sources disagree on whether is plausible, and this is what a zero local redundancy reports. Following (Ince, 2017a), such conflicting contributions are not counted as redundant; the information is assigned to the unique and synergistic atoms, and the local atoms still sum to the local mutual information. Pointwise PID describes what a particular realization conveys, and explicitly accepts zero or negative local values (Finn & Lizier, 2018; Liardi et al., 2026). The global narrowing noted by the reviewer is a distribution-level property and, as the reviewer acknowledges, is recovered when averaging over .
We believe these local conflicts are informative rather than a defect: multimodal interactions vary across samples, and this variation is useful both for analysis (Yang et al., 2025) and for learning (Yang et al., 2026). Accordingly, we keep the pointwise atoms signed throughout. In our human-attribution study (Sec. 4.5), their averages agree with human ratings of multimodal interactions (average Pearson , vs. 0.586 for LSMI), including on datasets where cross-modal incongruity is central (sarcasm in MUStARD: 0.831 vs. 0.506; sentiment in MOSEI: 0.512 vs. 0.078).
Reference. Yang, Z., Wei, Y., Ni, H., Xu, Z., & Hu, D. (2026). Information-Theoretic Decomposition for Multimodal Interaction Learning. CVPR 2026.
Liardi et al. 2026, The mathematical landscape of partial information decomposition: A comprehensive review of properties and measures. https://arxiv.org/abs/2603.06678.
Q1: Line 054.
Response: Line 054 actually refers to the OR gate, not XOR. OR and AND are equivalent up to relabeling of inputs and output (OR(x1,x2)=¬AND(¬x1,¬x2)), so both carry mechanistic redundancy when the inputs are independent. XOR, in contrast, is purely synergistic, as our experiments confirm (Sec. 4.3). Following the original example of Harder et al. (2013), we now mention both OR and AND.
Q2–Q3: Fragility of Gács–Körner and the role of its approximation.
Response: We thank the reviewer for this important point. We address the two problems separately: what the measure of Griffith et al. (2014) quantifies, and its fragility.
Being demanding is desirable for source redundancy. Griffith et al. (2014) define redundancy as I∧=maxQI(Q;Y) over variables Q that are deterministic functions of each source, i.e. the information about Y carried by the Gács–Körner common variable. As a measure of total redundancy, this is indeed too demanding. We use it differently. In our framework, total redundancy R is given by the average of pointwise redundancies (Eq. 3), and the supervised Gács–Körner quantity (Eq. 6) only measures its source part, Rsrc: the information about Y that the modalities actually share. For this purpose, being demanding is exactly what we want: source redundancy should count only the information that both modalities actually contain. Requiring a shared variable that captures Y guarantees this, and leaves out everything else, such as information seen by only one modality, or overlap that comes from the prediction rule alone, as in the OR gate where the two inputs share nothing. The latter is captured separately by Rmech=R−Rsrc.
Fragility: why we estimate a relaxed version. We agree that requiring Q to be an exact function of each source makes it fragile: I∧ vanishes as soon as the distribution has full support, so an arbitrarily small amount of noise drops it to zero. No estimator can recover such a discontinuous quantity from finite noisy samples, and continuity is considered essential for empirical PID (Liardi et al., 2026, Sec. 4.3). SupGK therefore relaxes the hard constraint into a penalty λE∥z1R−z2R∥22 (similarly to Kleinman et al., 2023), and recovers it as λ→∞. This relaxation is faithful where the strict quantity is meaningful and robust where it collapses. In our mixture benchmark (new Sec. 4.4), the modalities share a bit C but are observed with Gaussian noise. The strict quantity is therefore zero, although the modalities clearly share C, whereas SupGK successfully recovers the expected value I(C;Y)=H(C), i.e. the value of the strict quantity on the noise-free data.