acceptodds
Under review as a conference paper at ICLR 2027

Source vs. Mechanistic Redundancy in Pointwise Partial Information Decomposition

Abstract

We introduce a framework for pointwise partial information decomposition (PID) that estimates sample- and distribution-level redundancy, uniqueness, and synergy between two modalities and a target. We define an -inspired pointwise redundancy measure that is computed directly from predictive models, avoiding explicit high-dimensional input-density estimation. We further introduce a supervised Gács–Körner-inspired objective for learning target-predictive shared representations across modalities. This enables us to decompose predictive redundancy into source redundancy, arising from information shared between the modalities, and mechanistic redundancy, arising when distinct modality-specific mechanisms provide overlapping predictive information. Across analytically tractable synthetic benchmarks, controlled multimodal tasks, and clinical data, the framework improves pointwise estimation accuracy, recovers known source–mechanistic structure, and yields interpretable multimodal information-sharing patterns.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
partial information decomposition, multimodal learning, representation learning, pointwise information, redundancy, synergy, Gács–Körner common information, multimodal interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ye8LOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a way to separate the quantification of mechanistic and source redundancy. However, it has some serious flaws.

Critical strengths

The authors address an unsolved issue, but the solution proposed is not sound.

Critical weaknesses

The proposed method is not really able to quantify mechanistic and source redundancy separately, and on top has issues with the definition of unique information and the interpretation of local values.

Questions for authors

1- line 054054: mechanistic redundancy appears in the AND gate, not the XOR gate.

2-The redundancy measure of Griffith 2014 is known to have substantial problems, being too demanding in terms of what common information is. For example, it vanishes when the joint distribution P(X1...XnY)P(X_1...X_n Y ) has full support, hence for the majority of distributions. This also render sit brittle to noise. An additional problem is that the approximation of GK used in this paper may actually help to address this fragility of the GK, but conceptually it is not desirable that the method works precisely when it does not approximate GK too faithfully. The authors do comment on the fragility of the GK measure, but not on this consequence for the role of its approximation.

3-Related to the above, if the proposed approach is to work when departing to some degree from a strict GK estimation, then the exact valid range of $\lambda $ is unknown.

4- There is an interpretation problem of redundancy at the sample level: consider definition of Eq 3 and the case of p(Y)=N(0,4)p(Y)= N(0,4), p(Y∣x1)=N(0,1)p(Y|x_1)=N(0,1) and p(Y∣x2)=N(0,2)p(Y|x_2)= N(0,2), for some Xi=xiX_i =x_i. i1(y)=log2−3y2/8i_1(y)= log2 -3y^2/8, and i2(y)=0.5log2−y2/8i_2(y)= 0.5log2 - y^2/8. Therefore, there is an interval in which their signs differ. It is hard to interpret the consequence of this as a sample-PID with no redundancy in that interval, simply because what you do have is that across the whole distribution of YY x1x_1 and x2x_2 lead to a consistent narrowing of the variance. That narrowing is a global property that happens across all YY. The sample-PID semantics are hard to interpret locally in the interval where zero values are obtained, but they do lead to a consistent redundancy value when averaged across P(Y∣x1,x2)P(Y|x_1,x_2). So I think there is a problem in the interpretability of sample-level PIDs.

5- In Eq 5 and 6 pick ∣∣22||^2_2 or ∣∣2||^2, but be consistent.

6-In Eq 5 and 6 the meaning of C(i)C(i) and R(i)R(i) has not been described. The support of ii is not given. You need an explanation of i=1,...Ni=1,...N and what this means in terms of samples.

7-In Eq 7, even if z1z_1 is close to z2z_2, that does not enforce p(Y∣z1)p(Y|z_1) to be close to p(Y∣z2)p(Y|z_2). A clear example is to define YY with values −1,1-1,1 and Z1=δYZ_1 = \delta Y, Z2=−δYZ_2 = -\delta Y. For δ\delta limit to 0, Z1Z_1 and Z2Z_2 are always infinitesimally close, and yet P(Y=1∣Z1=δ)=1P(Y=1|Z_1=\delta)=1 and P(Y=1∣Z2=δ)=0P(Y=1|Z_2=\delta)=0. So similarity in the values of ziz_i does not imply alignment of the distributions p(Y∣zi)p(Y|z_i). This is different than the single ZZ Griffith et al approach, in that case really there is a single variable accounting for a single common p(Y∣z)p(Y|z).

8-The role of the shared prediction head is crucial to attenuate problems of the sort of point 7. However, this is not at all described when introduced. Even more, it is misleading that the use of subscripts is not consistent. the encoder ff should be source specific, fif_i, while hh is not source specific, isn't it? Only the common prediction head can constraint the misalignment of the type of point 7.

9-In fact, it is unclear from the main paper what is the role of the common prediction head. If ZiZ_i are trained previously to using the common prediction head, that does not attenuate the potential misalignment of the two representations and the fact of having substantially different p(Y∣zi)p(Y|z_i) even with very similar ziz_i.

10-The mapping from Eq 6 to Eq 7 is very nontrivial, the optimized objective function changes. The first term in Eq 7 it is misleading that it corresponds to H(Z∣Y)↓H(Z|Y)\downarrow, it is only monotonically paired. The second term uses Renyi entropy.

11-Not only the mapping from Eq 6 to 7 is nontrivial in terms of the mathematical derivation, it fundamentally alters the objective of the optimization. This is not well stated. When p(Y∣zi)p(Y|z_i) is noisy and within-class structure exists, minimizers of Eq 7 do not necessarily lead to good maximizers of Eq 6.

12-Related to the above, In Eq 6 the contribution of classes and their matched or unmatched predictions is weighted by their occurrence. In Eq 7 the normalization with ∣P|P| and ∣N∣|N| reweights these contributions in a way that is not sufficiently noticed in terms of its implications.

13-There are multiple definitions of redundancy that appear inconsistent, and this is not clarified. Eq 3 and 41 follow the sign criterion, Eq 8 does not. Is it done on purpose that the sign criterion is used for total redundancy and not for source redundancy?

14-The necessity of the reassignment in line 226 is mathematically convenient, but conceptually indicates a substantial flaw. The problem is that data processing inequality does not hold pointwise, and hence even if ziz_i is derived from xix_i, the info can be higher. This complicates even more the semantics of pointwise PIDs.

15- There is some conceptual problem between the fact that nonnegativity of the mechanistic information is enforced in line 226 and the fact that by doing so rr is updated to a value rsourcer_{source} that can be higher, leading to smaller unique information. In fact you can even have rsource>i(xi;y)r_{source}> i(x_i;y), so that you create a negative unique. This asymmetry between enforcing nonnegativity of the mechanistic redundancy and creating negativity of the unique is hard to swallow. If the negative unique is to be taken as unique misinformation, why not having redundant mechanistic misinformation?

16-The framework inherits a problem from the Williams & Beer formulation that additionally affects the interpretability of mechanistic redundancy. This is the lack of fulfillment of the identity property. Consider Y=(R0,U1,U2)Y= (R_0, U_1,U_2), X1=(R0,U1)X_1 = (R_0, U_1), and X2=(R0,U2)X_2= (R_0,U_2), where all three variables R0R_0, U1U_1, and U2U_2 have 1 bit. It is well known that IminI_{min} quantifies 'same amount of information' as redundancy, even if the information is about a different variable and the variables are independent. This affects the separation of mechanistic and source redundancy. With the strict GK approach of Griffith et al 2014, in this case R0R_0 can be retrieved functionally from both sources, and hence Rsource=1R_{source} =1. But the total redundancy is R=2R=2, because of the 'same amount of info issue'. This forces Rmech=1R_{mech}=1, despite the fact that there is no common mechanism beyond the common R0R_0. When using the procedure of Eq 7 to relax GK, RsourceR_{source} may capture more than one bit, up to two bits. In that case, RmechR_{mech} may be zero, but the quantification of the total redundancy is still inadequate, if it is accepted that redundancy should not correspond to simply 'same amount of info'.

12 comments

Q1: Line 054.

Reviewer ye8L: "line 054: mechanistic redundancy appears in the AND gate, not the XOR gate."

Response: Line 054 actually refers to the OR gate, not XOR. OR and AND are equivalent up to relabeling of inputs and output (OR(x1,x2)=¬AND(¬x1,¬x2)\text{OR}(x_1, x_2) = \lnot \text{AND}(\lnot x_1, \lnot x_2)), so both carry mechanistic redundancy when the inputs are independent. XOR, in contrast, is purely synergistic, as our experiments confirm (Sec. 4.3). Following the original example of Harder et al. (2013), we now mention both OR and AND.

Q2–Q3: Fragility of Gács–Körner and the role of its approximation.

Reviewer ye8L: "The redundancy measure of Griffith 2014 is known to have substantial problems, being too demanding in terms of what common information is. For example, it vanishes when the joint distribution has full support, hence for the majority of distributions. This also renders it brittle to noise. An additional problem is that the approximation of GK used in this paper may actually help to address this fragility of the GK, but conceptually it is not desirable that the method works precisely when it does not approximate GK too faithfully. The authors do comment on the fragility of the GK measure, but not on this consequence for the role of its approximation."

Response: We thank the reviewer for this important point. We address the two problems separately: what the measure of Griffith et al. (2014) quantifies, and its fragility.

Being demanding is desirable for source redundancy. Griffith et al. (2014) define redundancy as I∧=max⁡QI(Q;Y)I^\wedge = \max_Q I(Q;Y) over variables QQ that are deterministic functions of each source, i.e. the information about YY carried by the Gács–Körner common variable. As a measure of total redundancy, this is indeed too demanding. We use it differently. In our framework, total redundancy RR is given by the average of pointwise redundancies (Eq. 3), and the supervised Gács–Körner quantity (Eq. 6) only measures its source part, RsrcR_{\text{src}}: the information about YY that the modalities actually share. For this purpose, being demanding is exactly what we want: source redundancy should count only the information that both modalities actually contain. Requiring a shared variable that captures Y guarantees this, and leaves out everything else, such as information seen by only one modality, or overlap that comes from the prediction rule alone, as in the OR gate where the two inputs share nothing. The latter is captured separately by Rmech=R−RsrcR_{\text{mech}} = R - R_{\text{src}}.

Fragility: why we estimate a relaxed version. We agree that requiring QQ to be an exact function of each source makes it fragile: I∧I^\wedge vanishes as soon as the distribution has full support, so an arbitrarily small amount of noise drops it to zero. No estimator can recover such a discontinuous quantity from finite noisy samples, and continuity is considered essential for empirical PID (Liardi et al., 2026, Sec. 4.3). SupGK therefore relaxes the hard constraint into a penalty λE∥z1R−z2R∥22\lambda \mathbb{E} \Vert z_1^R - z_2^R \Vert^2_2 (similarly to Kleinman et al., 2023), and recovers it as λ→∞\lambda \to \infty. This relaxation is faithful where the strict quantity is meaningful and robust where it collapses. In our mixture benchmark (new Sec. 4.4), the modalities share a bit CC but are observed with Gaussian noise. The strict quantity is therefore zero, although the modalities clearly share CC, whereas SupGK successfully recovers the expected value I(C;Y)=H(C)I(C;Y) = H(C), i.e. the value of the strict quantity on the noise-free data.

Reviewer ye8L (Q3): "Related to the above, if the proposed approach is to work when departing to some degree from a strict GK estimation, then the exact valid range of lambda is unknown."

Choice of λ\lambda (Q3). We agree that no exact theoretical range for λ\lambda can be given, since the relative scale of the loss terms depends on the data. However, the valid range is bounded by two identifiable failure modes, corresponding to the two extremes of the relaxation. As λ→0\lambda \to 0, there is no constraint, and private information leaks into RsrcR_{\text{src}}. As λ→∞\lambda \to \infty, SupGK approaches the strict quantity and inherits its fragility: on noisy, high-dimensional data, shared information is suppressed (as observed on AV-MNIST, App. M.1). We tested this on three examples with known RsrcR_{\text{src}}:

True RsrcR_{\text{src}}λ=0\lambda=0λ=1\lambda=1λ=10\lambda=10λ=25\lambda=25λ=50\lambda=50λ=100\lambda=100
Two-bit copy (no shared info)0.0000.2010.2800.0560.0000.0000.000
Partial source (QjeU)0.6930.6870.6830.6910.6910.6910.691
Q16 example0.6930.7320.7900.6910.6910.6910.691

Table R1. Estimated RsrcR_{\text{src}} (nats, mean over 3 seeds; std ≤ 0.001 for λ≥25\lambda \ge 25).

Importantly, the right value can be identified without access to the true decomposition. Leakage has a visible symptom: when private information enters RsrcR_{\text{src}}, the source-redundancy estimates of the two modalities disagree. Our criterion therefore selects the smallest λ\lambda at which cross-modal agreement is reached (App. M.1); choosing the smallest such value avoids the over-alignment regime, in which further alignment only suppresses shared information. In the two-bit copy and in the mixture without shared bit, the cross-modal MAE is still 0.07–0.21 nats at λ=10\lambda = 10 and drops to 0.000 at λ=25\lambda = 25 (3 seeds), so the criterion selects λ=25\lambda = 25, where Rsrc=0R_{\text{src}} = 0. When shared information is present (Q16 example), agreement is already reached at λ=10\lambda = 10 (MAE 0.000), and the criterion selects λ=10\lambda = 10, as it does on AV-MNIST. On the Huntington's disease data, it selects λ=25\lambda = 25, and source redundancy changes only from 29.4% to 29.5% between λ=10\lambda = 10 and 25. We have added this discussion to App. M.1 (see also App. O, P).

Response: We thank the reviewer for this example, which we verified: for 1.85<y2<2.771.85 < y^2 < 2.77, i1(y)<0<i2(y)i_1(y) < 0 < i_2(y), so the local redundancy of Eq. 3 is zero. This zero has a clear meaning. Both sources narrow the distribution, but by different amounts: in this interval, yy already lies in the tail of the tighter p(y∣x1)p(y \mid x_1) while still in the bulk of the wider p(y∣x2)p(y \mid x_2). For this particular outcome, the two sources disagree on whether yy is plausible, and this is what a zero local redundancy reports. Both sources narrow the distribution, but by different amounts: in this interval, yy already lies in the tail of the tighter p(y∣x1)p(y \mid x_1) while still in the bulk of the wider p(y∣x2)p(y \mid x_2). For this particular outcome, the two sources disagree on whether yy is plausible, and this is what a zero local redundancy reports. Following IccsI_{\text{ccs}} (Ince, 2017a), such conflicting contributions are not counted as redundant; the information is assigned to the unique and synergistic atoms, and the local atoms still sum to the local mutual information. Pointwise PID describes what a particular realization conveys, and explicitly accepts zero or negative local values (Finn & Lizier, 2018; Liardi et al., 2026). The global narrowing noted by the reviewer is a distribution-level property and, as the reviewer acknowledges, is recovered when averaging over p(y∣x1,x2)p(y \mid x_1, x_2).

We believe these local conflicts are informative rather than a defect: multimodal interactions vary across samples, and this variation is useful both for analysis (Yang et al., 2025) and for learning (Yang et al., 2026). Accordingly, we keep the pointwise atoms signed throughout. In our human-attribution study (Sec. 4.5), their averages agree with human ratings of multimodal interactions (average Pearson r=0.733r = 0.733, vs. 0.586 for LSMI), including on datasets where cross-modal incongruity is central (sarcasm in MUStARD: 0.831 vs. 0.506; sentiment in MOSEI: 0.512 vs. 0.078).

Reference. Yang, Z., Wei, Y., Ni, H., Xu, Z., & Hu, D. (2026). Information-Theoretic Decomposition for Multimodal Interaction Learning. CVPR 2026.

Liardi et al. 2026, The mathematical landscape of partial information decomposition: A comprehensive review of properties and measures. https://arxiv.org/abs/2603.06678.

Reviewer_QjeUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a tractable estimator for pointwise PID that requires predictive models. It distinguishes mechanistic redundancy from source redundancy and introduces a supervised Gács–Körner objective (SupGK) for practical estimation. It validates its proposed methods on a couple of synthetic datasets and applies them to understand Huntington’s disease.

Critical strengths

The question of separating redundant, unique, and synergistic information, as well as separating source versus mechanistic synergistic information, is meaningful and important for understanding multimodal interpretability. The proposed estimations are easy to implement. Different evaluations have been performed across synthetic datasets and a disease application.

Critical weaknesses

The proposed definition of pointwise redundancy is inspired by IminI_{min}, which has been discussed as conflating unique information with redundant information (Harder et al., 2013). And since the authors define Rmech=R−RsourceR_{mech} = R - R_{source}, the estimate of RmechR_{mech} may inherit this concern and not capture mechanistic redundancy (see question 1). Additionally, it’s important to understand how SupGK would perform in a partial-source setting. The current SupGK seems to have a conflicting optimization goal in this setting (see question 2). Therefore, it’s unclear at this stage how to interpret the results from SupGK.

Questions for authors
  1. Consider the classical two-bit copy problem in [1]; RmechR_{mech} will instead capture the unique information in this case. Then how should we interpret RmechR_{mech}?

  2. It’s important to understand how SupGK would perform in a partial-source setting. For example, if we have X1=[C,A],X2=CX_1 = [C, A], X_2 = C, where CC and AA are i.i.d. Bernoulli (0.5), [] denotes concatenation, and Y=[C,A]Y = [C, A]. Assume we have sample 1: X1(1)=[0,0],X2(1)=0,Y(1)=[0,0]X_1^{(1)} = [0, 0], X_2^{(1)} = 0, Y^{(1)} = [0,0]. And sample 2: X1(2)=[0,1],X2(2)=0,Y(2)=[0,1]X_1^{(2)} = [0,1], X_2^{(2)} = 0, Y^{(2)} = [0,1]. The cross-modal alignment will encourage the representations g1(X1(1))=g1([0,0])g_1(X_1^{(1)}) = g_1([0,0]) and g2(X2(1))=g2(0)g_2(X_2^{(1)}) = g_2(0) to be close to each other, but at the same time, the different-class separation also encourages g1(X1(1))=g1([0,0])g_1(X_1^{(1)}) = g_1([0,0]) and g2(X2(2))=g2(0)g_2(X_2^{(2)}) = g_2(0) to be far from each other. Then, in this case, what would SupGK give? The current synthetic dataset (e.g., AV-MNIST) doesn’t include these scenarios since both modalities (images or spoken digits) can predict the whole Y (digit label).

  3. It’s important to add experiments to understand SupGK when source redundancy, mechanistic redundancy, and unique information all exist. And how SupGK’s performance changes across different proportions of them. Most of the current experiments focus on more extreme cases with 100% SRC or 0% SRC (Table 5). Although the authors include experiments on a mixture of UU and RR, and show that their methods split RR into 61.5% source and 38.5% mechanistic, the exact true split here is not available. A controlled experiment with known proportions of both redundancy sources would provide stronger evidence for the proposed method.

[1] Harder, M., Salge, C., & Polani, D. (2013). Bivariate measure of redundant information. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 87(1), 012130.

5 comments

We thank the reviewer for the precise counterexamples; working through them improved the paper. We ran each of them with our estimator and report the values below (nats).

Q1: Two-bit copy: how should RmechR_{\text{mech}} be interpreted?

Reviewer QjeU: "Consider the classical two-bit copy problem in [1]; R mech will instead capture the unique information in this case. Then how should we interpret R mech?"

Response: We thank the reviewer for this important example. In short, RmechR_{\text{mech}} does not capture unique information: it captures the redundancy that the base measure finds between sources that share nothing. In the two-bit copy, this is redundancy under Imin⁡I_{\min}, which our estimator takes inspiration from, and zero under identity-satisfying measures, such as BROJA.

The two-bit copy is a matter of convention. Which redundancy measure to use is still debated, because the properties one would like a measure to satisfy cannot all hold at once, even with two sources (Liardi et al., 2026). Each measure therefore reflects a choice. BROJA (Bertschinger et al., 2014) satisfies the identity property (Harder et al., 2013), which is why it reads the two-bit copy as one unique bit per source: each source is copied into its own component of Y=(X1,X2)Y = (X_1, X_2). Imin⁡I_{\min} (Williams & Beer, 2010) treats YY as a single categorical variable, as in classification, and asks, outcome by outcome, how much each source tells us about it: each source rules out half of the four outcomes, so both give the same amount of information about every outcome (redundancy), but only together do they identify it (through synergistic interaction). Both decompositions are consistent with the vanishing co-information I(X1;X2;Y)=0I(X_1;X_2;Y) = 0. Neither is wrong; they are two conventions about what counts as redundancy. The two readings also differ on synergy: both agree that identifying YY requires observing both sources, but Imin⁡I_{\min} counts this as synergy, whereas BROJA treats it as the union of two unique pieces of information, one per source. Note that neither convention excludes redundancy between independent sources: both find some in the OR and AND gates, which BROJA can reconcile with the identity property only by violating target monotonicity (Liardi et al., 2026, Prop. 8).

Why an Imin⁡I_{\min}-based measure. Imin⁡I_{\min} is defined outcome by outcome and only requires p(y∣xm)p(y \mid x_m) and p(y)p(y), so it can be evaluated per sample from standard unimodal predictors. BROJA, in contrast, is defined by an optimization over joint distributions, which in high dimension can only be approximated at the distribution level (e.g., BATCH; Liang et al., 2023). Since our goal is a pointwise decomposition, we build on the Imin⁡I_{\min} principle for our Eq. 3. Empirically, our estimator recovers the Imin⁡I_{\min} decomposition of the two-bit copy: R=0.676±0.002R = 0.676 \pm 0.002 and S=0.679±0.004S = 0.679 \pm 0.004 nats (vs. log⁡2=0.693\log 2 = 0.693), with U1,U2≈0.016U_1, U_2 \approx 0.016 (3 seeds; new App. P).

Our source/mechanistic split is compatible with both conventions. On these examples, the two conventions disagree only on redundancy between independent sources: BROJA finds some in the OR and AND gates, and Imin⁡I_{\min} in the OR and AND gates and in the two-bit copy. Our source redundancy does not depend on this choice. Since the sources are independent in all three cases, SupGK returns Rsrc≈0R_{\text{src}} \approx 0; in the two-bit copy, RsrcR_{\text{src}} is exactly 00 at λ=25\lambda = 25, the value selected by our criterion (response to Reviewer ye8L, Q3), and for all larger values; with weaker alignment (λ=10\lambda = 10) it is 0.056±0.0430.056 \pm 0.043 nats. Whatever redundancy the base measure finds is therefore attributed to RmechR_{\text{mech}}: about log⁡2\log 2 under Imin⁡I_{\min}, and 00 under BROJA. The convention-dependent part is isolated in RmechR_{\text{mech}}, while RsrcR_{\text{src}} captures only the redundancy that comes from information shared between the modalities. We have revised Sec. 3 to define RmechR_{\text{mech}} explicitly as this residual and to state that its meaning follows the base measure.

Q2: Partial-source setting.

Reviewer QjeU: "It's important to understand how SupGK would perform in a partial-source setting. For example, if we have X1=[C,A]X_1 = [C, A], X2=CX_2 = C, where CC and AA are i.i.d. Bernoulli(0.5), [ ] denotes concatenation, and Y=[C,A]Y = [C, A]. Assume we have sample 1: X1(1)=[0,0]X_1^{(1)} = [0, 0], X2(1)=0X_2^{(1)} = 0, Y(1)=[0,0]Y^{(1)} = [0, 0]. And sample 2: X1(2)=[0,1]X_1^{(2)} = [0, 1], X2(2)=0X_2^{(2)} = 0, Y(2)=[0,1]Y^{(2)} = [0, 1]. The cross-modal alignment will encourage the representations g1(X1(1))=g1([0,0])g_1(X_1^{(1)}) = g_1([0, 0]) and g2(X2(1))=g2(0)g_2(X_2^{(1)}) = g_2(0) to be close to each other, but at the same time, the different-class separation also encourages g1(X1(1))=g1([0,0])g_1(X_1^{(1)}) = g_1([0, 0]) and g2(X2(2))=g2(0)g_2(X_2^{(2)}) = g_2(0) to be far from each other. Then, in this case, what would SupGK give? The current synthetic dataset (e.g., AV-MNIST) doesn't include these scenarios since both modalities (images or spoken digits) can predict the whole YY (digit label)."

Response: We thank the reviewer for this sharp example. The reviewer is right that the terms of Eq. 7 pull in opposite directions here: g1([0,0])g_1([0,0]) is asked to be close to g2(0)g_2(0) (alignment with sample 1) and far from the same point g2(0)g_2(0) (separation from sample 2, which has a different label), since X2=CX_2 = C cannot distinguish the two samples. This tension cannot inflate RsrcR_{\text{src}}, however, because Eq. 8 takes, for each sample, the minimum pointwise mutual information over the two modalities (i.e., the worst of the two predictions), so rsrc≤i(z2R;y)r_{\text{src}} \le i(z_2^R; y). On average, Rsrc≤I(Z2R;Y)≤I(C;Y)=log⁡2R_{\text{src}} \le I(Z_2^R; Y) \le I(C; Y) = \log 2, the correct value, by the data processing inequality, since z2Rz_2^R only sees X2=CX_2 = C.

We ran exactly this example (3 seeds). The ground truth is Rsrc=R=U1=0.693R_{\text{src}} = R = U_1 = 0.693 nats (the shared bit CC is redundant, the private bit AA is unique to X1X_1) and U2=S=Rmech=0U_2 = S = R_{\text{mech}} = 0. The table reports the information each SupGK representation carries about YY, and the resulting RsrcR_{\text{src}}:

Trueλ=1\lambda = 1λ=10\lambda = 10
Information in z1Rz_1^R (sees CC and AA)0.6930.8860.692
Information in z2Rz_2^R (sees CC only)0.6930.6850.692
RsrcR_{\text{src}} (minimum)0.6930.6830.691

In Eq. 7, z1Rz_1^R is only compared with z2Rz_2^R points, which here depend only on CC. The loss therefore never rewards encoding AA in z1Rz_1^R, but with weak alignment (λ=1\lambda = 1) it does not prevent it either: z1Rz_1^R is only loosely tied to z2Rz_2^R, and this leaves room for AA. This is why z1Rz_1^R carries more information than the true value. This does not affect RsrcR_{\text{src}}, because the minimum selects z2Rz_2^R, which only sees CC. As λ\lambda increases, the alignment term forces z1Rz_1^R onto z2R=g2(C)z_2^R = g_2(C), so z1Rz_1^R can no longer encode AA. This is exactly the Gács–Körner solution, in which both representations keep the shared bit and drop the private one. At λ=10\lambda = 10, the full decomposition is recovered: Rsrc=0.691±0.001R_{\text{src}} = 0.691 \pm 0.001, R=0.692R = 0.692, U1=0.693U_1 = 0.693, U2=S=0.000U_2 = S = 0.000, Rmech=0.001R_{\text{mech}} = 0.001.

More generally, the new mixture benchmark (response to Reviewer L8Vf, W2/W3, and Q3 below) is a partial-source setting at scale: each modality observes the shared bit CC together with private bits the other does not see, and RsrcR_{\text{src}} is recovered accurately across all mixtures.

Q3: Coexisting source, mechanistic and unique information.

Response: We added the requested experiment (new App. O). It extends the controlled benchmark of our response to Reviewer L8Vf (W2/W3) with a unique component.

Data. As before (see Reviewer L8Vf (W2/W3) response), both modalities observe a shared bit C∼Bernoulli(pc)C \sim \text{Bernoulli}(p_c), and each modality has mm private bits A1A_1 and A2A_2 (Bernoulli(0.5)\text{Bernoulli}(0.5)). We add one more private bit U∼Bernoulli(0.5)U \sim \text{Bernoulli}(0.5), seen only by modality 1. The inputs are X1=[C,A1,U]X_1 = [C, A_1, U] and X2=[C,A2]X_2 = [C, A_2], with Gaussian noise (σ=0.1\sigma = 0.1), and the target is

Y=(C,OR(A1,1,A2,1),…,OR(A1,m,A2,m),U).Y = \left(C, \text{OR}(A_{1,1}, A_{2,1}), \dots, \text{OR}(A_{1,m}, A_{2,m}), U\right).

Each component of YY creates one kind of information: CC is shared by both modalities (source redundancy), the OR gates combine independent private bits (mechanistic redundancy), and UU is only available to modality 1 (unique information).

Exact ground truth. Since all variables are discrete with a known distribution, every quantity is computed exactly by summing over all configurations of (C,A1,A2,U)(C, A_1, A_2, U). As before, Rsrc=H(C)R_{\text{src}} = H(C), which follows from the structure of the data alone. RR is the population value of Eq. 3, and U1U_1, U2U_2 and SS follow from Eq. 1 using the exact mutual informations I(X1;Y)I(X_1;Y), I(X2;Y)I(X_2;Y) and I(X1,X2;Y)I(X_1,X_2;Y).

(pc,m)(p_c, m)True % sourceEstimated % sourceTrue RsrcR_{\text{src}}Estimated RsrcR_{\text{src}}True U1U_1Estimated U1U_1
(–, 1)0.055.6 ± 9.30.0000.115 ± 0.0180.5920.690 ± 0.011
(–, 1), with λ=25\lambda = 250.00.0 ± 0.00.0000.000 ± 0.0000.5920.690 ± 0.011
(0.05, 3)24.025.3 ± 0.40.1990.185 ± 0.0050.7100.691 ± 0.017
(0.05, 1)39.848.3 ± 1.30.1990.192 ± 0.0040.6090.668 ± 0.006
(0.10, 2)43.044.4 ± 0.50.3250.317 ± 0.0050.6930.668 ± 0.005
(0.20, 2)54.557.0 ± 0.70.5000.493 ± 0.0070.7070.695 ± 0.005
(0.50, 2)63.767.2 ± 0.50.6930.690 ± 0.0000.7290.733 ± 0.006
(0.50, 1)76.378.6 ± 0.70.6930.691 ± 0.0000.6930.692 ± 0.006
(0.50, 0)100.0100.0 ± 0.00.6930.691 ± 0.0000.6930.689 ± 0.001

(Nats; mean ± std over 3 seeds; λ=10\lambda = 10, as in our main experiments, unless stated; (–, 1): no shared bit.)

Results. The estimator recovers all three kinds of information. Whenever the modalities share information, RsrcR_{\text{src}} is accurate (mean absolute error 0.006 nats) and the source fraction tracks the ground truth (mean absolute error 2.8 points). Unique information is attributed to the right modality: U2U_2 stays close to its true value of about 0 (estimated 0.000–0.062), and U1U_1 tracks its true value, with a slight overestimation in some settings with OR gates. Both small biases, in the source fraction and in U1U_1, have the same origin: total redundancy is slightly underestimated, so the source fraction Rsrc/RR_{\text{src}} / R is inflated, and since U1=I(X1;Y)−RU_1 = I(X_1;Y) - R, the missing redundancy is counted as unique information. RsrcR_{\text{src}} itself is not affected.

Setting without shared information. Here our selection criterion chooses λ=25\lambda = 25, not 10: at λ=10\lambda = 10 the two modalities' source-redundancy estimates still disagree (MAE 0.21 nats), which signals that private information is leaking into RsrcR_{\text{src}} (0.115 nats instead of 0), and agreement is reached only at λ=25\lambda = 25 (MAE 0.000), where Rsrc=0.000R_{\text{src}} = 0.000. This choice was made using our selection criterion (App. M) and thus uses no knowledge of the true decomposition (response to Reviewer ye8L, Q3). With the selected value, the mean absolute error on the source fraction over all settings is 2.4 points.

We hope these clarifications and experiments address the reviewer's concerns. If so, we would be grateful if the reviewer would consider updating their score.

Reviewer_L8VfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a framework for estimating pointwise Partial Information Decomposition and further distinguishing two sources of predictive redundancy: source redundancy and mechanistic redundancy. Source redundancy arises from target-relevant information shared between modalities, while mechanistic redundancy occurs when distinct sources provide overlapping predictive information through the target-generating mechanism. This paper introduces a Supervised Gács–Körner objective to learn target-predictive representations that can be recovered from both modalities, thereby estimating source redundancy and assigning the remaining predictive redundancy to its mechanistic component.

Critical strengths

The authors propose a sample-level PID estimation method. The problem addressed is significant, and the estimation approach is both simple and logical. The concept of distinguishing between source redundancy and mechanistic redundancy is highly novel. The paper provides extensive experimental evidence to demonstrate improvements over existing methods.

Critical weaknesses
  1. Although the distinction between source redundancy and mechanistic redundancy offers an intriguing information-theoretic perspective, the paper does not clearly explain why this distinction is important or what limitations of traditional PID it addresses.

  2. Much of the experimental evaluation focuses on validating the standard PID components, providing limited new insights. In contrast, the evaluation of the source–mechanistic distinction is relatively limited. It is also unclear whether this distinction can provide practical benefits for tasks such as data filtering or model selection.

  3. Section 4.4 mainly evaluates extreme cases where redundancy is either entirely source-based or entirely mechanistic. It remains unclear whether the method can accurately estimate the relative contributions of the two components when they coexist.

Questions for authors

Could mechanistic redundancy be estimated directly, rather than as the difference between total redundancy and source redundancy?

To what extent do the consistency corrections affect the reported PID results?

Could the authors improve the readability of Table 8?

5 comments

We thank the reviewer for recognizing the novelty of distinguishing source from mechanistic redundancy and the simplicity of the estimator. We address each point below.

W1: Why the distinction matters and which limitation of standard PID it addresses.

Reviewer L8Vf: "Although the distinction between source redundancy and mechanistic redundancy offers an intriguing information-theoretic perspective, the paper does not clearly explain why this distinction is important or what limitations of traditional PID it addresses."

Response: Standard PID returns a single redundancy atom, which conflates two situations with different meanings. In the AND/OR gates, the inputs are independent (I(X1;X2)=0I(X_1;X_2)=0), yet redundancy is non-zero: both inputs inform YY in overlapping ways only because of the prediction rule. In AV-MNIST, by contrast, spoken and written digits encode the same concept, and redundancy arises from information the modalities genuinely share. This limitation is well recognized: Harder et al. (2013) introduced the terms source and mechanistic redundancy, Pica et al. (2017) formalized a related split into source and non-source redundancy, and Ince (2017b) showed it can be quantified via a partial entropy decomposition. These approaches, however, are restricted to low-dimensional discrete variables, and no estimator separates the two components in high-dimensional multimodal data, let alone per sample. A recent review of PID (Liardi et al., 2026) also stresses that mechanistic redundancy, which can be non-zero even when the sources are independent, is still poorly understood and that it is "fundamental to investigate its significance". Our supervised Gács--Körner formulation addresses this gap: source redundancy is the target information carried by a representation recoverable from both modalities, f1(X1)≈f2(X2)=ZRf_1(X_1) \approx f_2(X_2) = Z^R, with Rsrc=I(ZR;Y)R src = I(Z^R;Y) (App. D), and mechanistic redundancy is the remainder.

The split matters because the two components support different conclusions: source redundancy indicates that the modalities measure a common underlying factor, whereas mechanistic redundancy indicates that independent factors lead to the same prediction, a notion closely related to degeneracy in biological systems (i.e., structurally different elements performing the same biological function; Tononi et al., 1999). Moreover, our source redundancy estimator comes with a shared representation ZRZ^R, which tells us not only how much information two modalities share, but which information. Our Huntington's disease application (Sec. 5) illustrates this. Speech is an emerging low-cost biomarker of the disease, but it is unclear whether a speech-based predictor of lexical fluency captures the striatal motor degeneration that causes dysarthria, or the higher-order cognitive decline that lexical fluency also measures. Aligning neuroanatomy to speech with SupGK keeps only the brain information that speech also carries, which yields a testable prediction: if speech mainly reflects motor degeneration, striatal regions should keep their predictive importance under alignment, while associative cortical regions should lose it. This is what we observe: the putamen and caudate retain 60--73% of their importance, whereas the posterior cingulate, angular gyrus and frontal operculum retain only 39--57%; randomly pairing speech and MRI recordings drops source redundancy to 0% and retention to 6%. A single redundancy atom would tell us that speech and neuroanatomy overlap, but not what that overlap consists of; the source component, through its shared representation, makes this question answerable. We have added a paragraph making this motivation explicit in the introduction.

Liardi et al. 2026, The mathematical landscape of partial information decomposition: A comprehensive review of properties and measures. https://arxiv.org/abs/2603.06678.

W2/W3: Evaluation when source and mechanistic redundancy coexist.

Reviewer L8Vf: "Much of the experimental evaluation focuses on validating the standard PID components, providing limited new insights. In contrast, the evaluation of the source–mechanistic distinction is relatively limited. It is also unclear whether this distinction can provide practical benefits for tasks such as data filtering or model selection. Section 4.4 mainly evaluates extreme cases where redundancy is either entirely source-based or entirely mechanistic. It remains unclear whether the method can accurately estimate the relative contributions of the two components when they coexist."

Response: We thank the reviewer for this suggestion, which addresses the main gap of the previous evaluation. We added a controlled benchmark in which source and mechanistic redundancy coexist in known proportions (new Sec. 4.4).

Data. Both modalities observe a shared bit C∼Bernoulli(pc)C \sim \mathrm{Bernoulli}(p_c), and each modality has mm private bits, A1A_1 and A2A_2 (Bernoulli(0.5)\mathrm{Bernoulli}(0.5)). The inputs are X1=[C,A1]X_1 = [C, A_1] and X2=[C,A2]X_2 = [C, A_2], with Gaussian noise (σ=0.1\sigma = 0.1), and the target is Y=(C,OR(A1,1,A2,1),…,OR(A1,m,A2,m))Y = \left(C, \mathrm{OR}(A_{1,1}, A_{2,1}), \dots, \mathrm{OR}(A_{1,m}, A_{2,m})\right), encoded as a single class label. The shared bit CC creates source redundancy. Each OR gate creates mechanistic redundancy: its two inputs are independent, yet observing a 1 in either modality reveals the output. Thus pcp_c sets the amount of shared information and mm the amount of mechanistic redundancy.

Exact ground truth. Since all variables are discrete, every quantity is computed exactly by enumerating all configurations of (C,A1,A2)(C, A_1, A_2). Source redundancy can be computed directly from how the data are generated: the only information shared by X1X_1 and X2X_2 is the bit CC, which is also part of YY, so Rsrc=H(C)R_{\mathrm{src}} = H(C). This value does not depend on our redundancy measure (Eq. 3) or on SupGK. Total redundancy RR is the population value of Eq. 3, and Rmech=R−RsrcR_{\mathrm{mech}} = R - R_{\mathrm{src}}. Our estimator, in contrast, only sees finite noisy samples.

(pc,m)(p_c, m)True % sourceEstimated % sourceTrue RsrcR_{\mathrm{src}}Estimated RsrcR_{\mathrm{src}}
(–, 1)0.00.0 ± 0.00.0000.000 ± 0.000
(0.05, 3)30.833.7 ± 0.60.1990.188 ± 0.004
(0.05, 1)48.553.7 ± 3.80.1990.202 ± 0.012
(0.10, 2)56.258.7 ± 1.10.3250.318 ± 0.004
(0.20, 2)68.573.0 ± 1.20.5000.494 ± 0.006
(0.50, 2)78.884.9 ± 1.00.6930.691 ± 0.000
(0.50, 1)94.298.7 ± 0.50.6930.692 ± 0.000
(0.50, 0)100.0100.0 ± 0.00.6930.692 ± 0.000

(Nats; mean ± std over 3 seeds; (–, 1): no shared bit.) RsrcR_{\mathrm{src}} is recovered almost exactly (mean absolute error 0.004 nats), including Rsrc=0R_{\mathrm{src}} = 0 when the modalities share nothing, and the estimated source fraction tracks the ground truth across the whole range (mean absolute error 3.2 points).

Practical benefits. Interaction estimates are already used in practice: dataset-level PID can guide model selection (Liang et al., 2023), and sample-level interactions enable redundancy-informed data partitioning, targeted distillation and interaction-aware ensembling (Yang et al., 2025), as well as improved multimodal learning (Yang et al., 2026). Our split refines these uses by indicating why modalities are redundant: because they measure a common underlying factor (source), or because independent factors lead to the same prediction (mechanistic). Moreover, source redundancy comes with a shared representation that localizes the redundant information, as in our Huntington's disease analysis (Sec. 5). We leave their evaluation to future work.

(Yang et al., 2026): Zequn Yang and Yake Wei and Haotian Ni and Zhihao Xu and Di Hu, Information-Theoretic Decomposition for Multimodal Interaction Learning, CVPR 2026.

Q1: Can mechanistic redundancy be estimated directly?

Reviewer L8Vf: "Could mechanistic redundancy be estimated directly, rather than as the difference between total redundancy and source redundancy?"

Response: We define mechanistic redundancy as a residual on purpose, following the treatment of non-source redundancy in Harder et al. (2013) and Pica et al. (2017): there is no independent operational definition of it, only of total redundancy and of its shared part. The residual keeps the decomposition exact (R=Rsrc+RmechR = R_{\mathrm{src}} + R_{\mathrm{mech}}) without adding another convention. A direct estimator is possible, e.g. the redundancy that remains given the source-redundant representation, r~mech=min⁡(i(x1;y∣zR), i(x2;y∣zR))\tilde r_{\mathrm{mech}} = \min\left(i(x_1;y \mid z^R),\, i(x_2;y \mid z^R)\right). However, since the pointwise minimum is not additive, R~mech\tilde R_{\mathrm{mech}} would be a different quantity from R−RsrcR - R_{\mathrm{src}}, and the decomposition would no longer sum to RR. We leave the study of such direct estimators to future work.

Q3: Readability of Table 8 (now Table 9).

Response: We thank the reviewer and have restructured Table 8 (now Table 9). Each scenario is now a separate block in which the ground-truth decomposition is listed first, directly above the estimates of each method, so every atom can be compared with its reference row by row. Scenarios are grouped by type (pure, two-component and three-component mixtures), the best Δ\Delta per scenario is in bold, and the fraction of redundancy attributed to source redundancy has its own column. The values are unchanged.

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

We thank all reviewers for their careful reading. We have uploaded a revised PDF, with all changes highlighted in blue. The main changes are:

  1. Controlled benchmarks with exact ground truth, in which source redundancy, mechanistic redundancy and unique information coexist in known proportions (new Sec. 4.4, App. O). Source redundancy is recovered with a mean absolute error of 0.004–0.006 nats (L8Vf, QjeU).
  2. The reviewers' own examples, run with our estimator: two-bit copy (App. P), partial-source setting, inverted representations, and the identity-property example (detailed in our replies) (QjeU, ye8L).
  3. Removal of the pointwise reassignment: pointwise mechanistic redundancy is now signed and treated symmetrically with unique misinformation (Sec. 3.2). The human-attribution study now uses no post-hoc correction for either method (Sec. 4.5) (ye8L, L8Vf).
  4. Selection of λ: our criterion picks the correct value without access to the true decomposition, including when the modalities share no information (App. M.1) (ye8L).
  5. Eq. 7 as a surrogate: it is now presented explicitly as a surrogate of Eq. 6, with its approximations and limitations stated (App. C) (ye8L).
  6. Motivation and clarity: a paragraph on why the source/mechanistic split matters, and clarified notation and roles of the shared head (Sec. 1, 3.2) (L8Vf, ye8L).

We would be glad to know whether these changes address your concerns, and to clarify any remaining point before the end of the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.