acceptodds
Under review as a conference paper at ICLR 2027

Redundancy and synergy in multivariate Gaussians via the Blackwell order

Abstract

The goal of the partial information decomposition (PID) is to quantify the redundant and synergistic information that multiple sources provide about a target. PID has many applications in machine learning, neuroscience, and other fields, but defining and computing it for high-dimensional continuous systems remains challenging. Here, we define a PID for multivariate Gaussian systems using the Blackwell order, which formalizes when one channel is more informative than another. We show that, among existing redundancy measures that apply to any number of sources, Blackwell redundancy is the only one that satisfies a natural set of desiderata. We also prove that Gaussian channels are optimal for extracting both redundant and union information, yielding an exact and efficient PID as well as an intuitive geometric interpretation in terms of nested ellipsoids. Our union information and synergy coincide with the well-known BROJA measures, and we derive closed-form expressions for both in the case of two sources. We demonstrate the scalability of our method on systems with up to a thousand dimensions or sources. Finally, we illustrate the approach on an optimal control problem, uncovering redundant and synergistic interactions between sensor and memory.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
information theory, partial information decomposition, redundancy, synergy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PLLFOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper derives an analytical solution of synergy and redundancy decompositions for multivariate Gaussian processes.

Critical strengths

The paper is conceptually sharp, with Theorems 1-3 containing the main results. All mathematical details are left for the Supplementary material. But the result is very strong, and I can't find any flaws in the derivations. The results are elegant by their generality and in the PID field are very relevant, providing analytical results that were missing. The results are likely to find broad application, since Gaussian approximations are common in practice.

Critical weaknesses

I think the paper is very strong and I do not have much to object.

Questions for authors

1-I think it would be enlightening to comment on the connection with the Gaussian Information Bottleneck (Chechik 2005). At first, given that Kolchinsky 2024 reformulated the Blackwell redundancy in terms of an information bottleneck, my first instinct was to compare the derivations here presented with the Gaussian bottleneck of Chechik, and see if it was superseded by it. It turns out, I think, that it is the opposite, apart from defining Gaussian redundancy one could use the tools developed in this paper to formulate more general versions of the Chechik results. Of course I do not expect the authors to do that here, but some comments on the connection, even if in the supplementary material, would be helpful, if anything to avoid my first reaction in the direction that some of the results might had overlapped with existing Gaussian information-bottleneck results.

2- In theorem 4, W∪∗=V−Tdiag(min(1,λ1),...,min(1,λdY))V−1W^*_{\cup} = V^{-T} diag(min(1,\lambda_1),...,min(1,\lambda_{d_Y}))V^{-1}, not only one λj\lambda_j

3-Theorem 4 would be easier to understand if writing: For C1≻0C_1\succ 0, C2⪰0C_2 \succeq 0 let VV and Λ=diag(λ1,...,λdY)\Lambda = diag(\lambda_1,...,\lambda_{d_Y}) solve the generalize eigenvalue problem VTC1V=IV^T C_1 V = I, VTC2V=ΛV^T C_2 V = \Lambda....

The condition C1≻0C_1 \succ 0 is needed to guarantee that you can define A=C1−1/2C2C1−1/2A= C_1^{-1/2}C_2C_1^{-1/2} such that by symmetry A=UΛUTA = U \Lambda U^T and VV can be defined as V=C1−1/2UV= C_1^{-1/2} U, such that VTC1V=IV^T C_1 V = I, VTC2V=ΛV^T C_2 V = \Lambda. The condition C2⪰0C_2 \succeq 0 is needed so that no negative eigenvalues exist, or otherwise W∪∗W^*_{\cup} would not be PSD. Note that singularities have been allowed for Theorem 2 and 3, although in Theorem 4 it would lead to an infinite I∪I_{\cup}.

So, this formulation would be clearer, but still it remains asymmetric in the role of C1C_1 and C2C_2. This is reflected in the conditions C1,≻0C_1,\succ 0, C2⪰0C_2 \succeq 0, and in the different roles they play in VTC1V=IV^T C_1 V = I, VTC2V=ΛV^T C_2 V = \Lambda.

A better slightly more general presentation can be used with S=C1+C2S = C_1 + C_2, requiring S≻0S\succ 0. Then with A=S−1/2C1S−1/2A=S^{-1/2}C_1S^{-1/2}, S−1/2C2S−1/2=I−AS^{-1/2}C_2S^{-1/2}=I-A, and for UU orthogonal $U^T A U=P = diag(p_1,\ldots,p_{d_Y})$. Then with V=S−1/2UV = S^{-1/2} U, we obtain VTC1V=PV^T C_1V=P and VTC2V=I−PV^T C_2V=I-P, and 0≤pj≤10\le p_j\le1.

Done in this way Eq 14 becomes symmetric in C1C_1 and C2C_2. You get W∪∗=V−Tdiag(min(p1,1−p1),...,min(pdY,1−pdY))V−1W^*_{\cup} = V^{-T} diag(min(p_1,1-p_1),...,min(p_{d_Y},1-p_{d_Y}) )V^{-1}. Eq 14 becomes then more transparent, you have $I_{\cup} = -\frac12\log|C_1+C_2| -\frac12\sum_j \log\min(p_j,1-p_j)$, so it is interpretable that in the spectral domain in each mode the smaller posterior is selected. Apart from being more interpretable, it is slightly more general S≻0S\succ 0 than requiring C1≻0C_1\succ 0, C2⪰0C_2 \succeq 0 or C2≻0C_2\succ 0, C1⪰0C_1 \succeq 0.

4-Note that in fact, for C1,C2⪰0C_1,C_2 \succeq 0, then S=C1+C2≻0S= C_1 + C_2\succ 0 happens if and only if ker⁡C1∩ker⁡C2={0}\ker C_1\cap\ker C_2=\{0\}, as can be seen from XT(C1+C2)x=0X^T (C_1+C_2) x=0, which happens iff XTCix=0X^T C_i x=0 for both. Therefore, the symmetric formulation with SS can be related to the considerations in line 250 above Eq (9).

1 comment

We thank the referee for the positive assessment of our work and for the detailed and useful suggestions.

Question 1: Connection to the Gaussian information bottleneck

We thank the referee for this suggestion, and we agree with the referee's intuition: our results are not superseded by those of Chechik et al. (2005). The Gaussian information bottleneck (IB) concerns a single source and trades off prediction against compression. Blackwell redundancy concerns several sources and places no cost on compression. The bottleneck formulation of Blackwell redundancy in Kolchinsky (2024) is also different, since it relaxes the requirement that the channel be produced from every source rather than adding a cost on compression.

Following the referee's suggestion, we added a new Sec. S2 that connects the two problems. There we define a rate-constrained version of Blackwell redundancy: this reduces to Blackwell redundancy when the rates are unconstrained, and to the IB problem when there is a single source (it also encompasses the multi-view IB of Huang et al. (2022)). We analyze this generalized formulation by extending our proof of Theorem 2, using an approach similar to the analysis of the vector Gaussian CEO problem by Uğur et al. (2020), which we now cite. As the referee anticipated, this provides an alternative proof of Gaussian optimality for the IB and related objectives. We mention this connection in Sec. 5.1 and refer to Sec. S2 for details.

Questions 2–4: Presentation of Theorem 4

We thank the referee for these suggestions, which led us to restate Theorem 4. We agree that the previous statement treated the two sources asymmetrically, and that the notation for the optimizer was unclear.

The revised theorem considers any invertible matrix VV that diagonalizes both conditional covariances, V⊤ΣY∣X1V=diag⁡(a1,…,adY)V^\top\Sigma_{Y|X_1}V=\operatorname{diag}(a_1,\ldots,a_{d_Y}) and V⊤ΣY∣X2V=diag⁡(b1,…,bdY)V^\top\Sigma_{Y|X_2}V=\operatorname{diag}(b_1,\ldots,b_{d_Y}). Union information is then achieved by a Gaussian channel with posterior covariance V−⊤diag⁡(min⁡(a1,b1),…,min⁡(adY,bdY))V−1V^{-\top}\operatorname{diag}(\min(a_1,b_1),\ldots,\min(a_{d_Y},b_{d_Y}))V^{-1}, now written with all of its diagonal entries, and it is given by

+\tfrac14\sum_{j}\left\lvert\ln\frac{a_j}{b_j}\right\rvert .$$ This statement is symmetric in the two sources. It also has the interpretation suggested by the referee: $a_j$ and $b_j$ are the posterior variances left by the two sources along common directions of the target, and the union channel keeps the smaller one in each direction. We now mention this interpretation after the theorem. The matrix $V$ is not unique, since rescaling its columns multiplies $a_j$ and $b_j$ by the same factor, which leaves the theorem unchanged. Different choices of scale recover the two earlier formulations. The referee's formulation corresponds to $V^\top(\Sigma_{Y|X_1}+\Sigma_{Y|X_2})V=I$, which gives $a_j=p_j$ and $b_j=1-p_j$. The formulation of the submitted version corresponds to $V^\top\Sigma_{Y|X_1}V=I$, which gives $a_j=1$ and $b_j=\lambda_j$. We state the theorem without fixing the scale, so that it contains both formulations. Regarding singular matrices, the revised theorem no longer assumes that the conditional covariances are positive definite: it states that $I_\cup=+\infty$ if any $a_j$ or $b_j$ vanishes. Theorem 3 was updated in the same way, so that Theorems 2–4 now all allow singular conditional covariances. The referee is also right that $C_1+C_2\succ0$ is equivalent to $\ker C_1\cap\ker C_2=\lbrace0\rbrace$, which is the condition for finite redundancy stated in Sec. 5.1.
Reviewer_eawqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper develops a new theory for partial information decomposition in multivariate Gaussian systems based on the Blackwell order. The central contribution is to show that Gaussian channels suffice to optimize both redundant and union information, allowing these quantities to be computed through convex optimization problems involving posterior covariance matrices. This is supposed to provide information on how an output Y is composed of information from different sources

Critical strengths

1/Elegant geometric interpretation. The connection between the Blackwell order, Loewner ordering of posterior covariances, and ellipsoid inclusion provides an intuitive interpretation of redundancy and union information.

2/Computational efficiency. The convex log-determinant formulations and the closed-form expression for two-source union information are appealing. Working in target space can substantially reduce computational cost, particularly when the source dimensions greatly exceed the target dimension.

Critical weaknesses

1/ What provides PID methods with respect to standard attibution methods. How this can be used for explainability purpose

2/Novelty relative to previous Gaussian PID methods needs sharper clarification. The underlying Blackwell PID framework is inherited from Kolchinsky (2022), while Gaussian PID optimization and Gaussian optimality for BROJA have already been investigated, notably by Venkatesh et al. (2023) and Zhao et al. (2025). The paper contributes new results, particularly for redundancy and multiple sources, but should more explicitly isolate the conceptual and mathematical advances over these approaches.

3/he framework assumes that Gaussian covariance matrices are known. In real applications, they must be estimated from finite samples, potentially in high-dimensional settings. The paper does not establish how covariance estimation errors propagate to redundancy and synergy estimates, or how stable the proposed methods are under noisy, ill-conditioned, or nearly singular covariance matrices. This is an important limitation for practical deployment. So no estimation is provided.

4/ Simulations are provided but how this could be used in real applications.

Questions for authors

1/ Statistical estimation: How sensitive are the redundancy and synergy estimates to errors in the estimated covariance matrices? Can the authors derive consistency results, perturbation bounds, or confidence intervals for the resulting PID quantities?

2/ Can the authors provide an application of this method

1 comment

We thank the referee for the comments. We respond to the weaknesses and questions in order.

Weakness 1: PID and attribution methods

We feel that attribution methods and the PID answer different questions. An attribution method assigns an importance score to each input. The PID instead describes how information about the target is distributed among multiple inputs: for instance, which information is shared by the inputs (redundancy) and which is available only when they are combined (synergy).

As an example, two inputs with the same importance score may carry the same redundant information about the target, in which case either one can be removed without loss. Alternatively, they may carry complementary information, which is available only when both are observed together. These two situations have different implications, e.g., for robustness to the loss of an input, but they cannot be distinguished using importance scores alone. In addition, the Blackwell PID identifies the content of the shared information, not only its amount (Secs. 5.1 and 5.2). For these reasons, we see the PID as a useful complement to attribution methods.

Weakness 2: Novelty relative to previous work

We agree that our contributions should have been stated more clearly. To address this, we added a paragraph on contributions and prior work to the Introduction. In brief:

  • Blackwell redundancy was previously considered only for discrete variables (Kolchinsky, 2022). It has not been studied for Gaussian systems, so all of our results on redundancy are new. To our knowledge, it is the first redundancy measure for multivariate Gaussian systems that satisfies a set of natural conditions (Sec. 4).
  • Blackwell union information and synergy coincide with the BROJA measures, which were previously studied for two Gaussian sources (Venkatesh et al., 2023; Zhao et al., 2025). We prove Gaussian optimality for any number of sources, reduce the computation to an optimization over a single dY×dYd_Y\times d_Y matrix, and derive a closed-form expression for two sources.
  • We prove continuity and consistency results, described below. For the Gaussian BROJA PID, consistency was previously reported in numerical experiments, but not proven.

Weakness 3 and Question 1: Statistical estimation

We thank the referee for this question, which led us to add new results on continuity and estimation.

We now prove that union information and synergy are continuous functions of the covariance matrices (Proposition 2 in Sec. 5.2). The same holds for redundancy, as long as all sources except possibly one have full rank, rank⁡ΣYXi=dY\operatorname{rank}\Sigma_{YX_i}=d_Y, and more generally at almost all covariance matrices (Proposition 1 in Sec. 5.1). Under these conditions, the PID can be consistently estimated from data, with error of order N−1/2N^{-1/2} for NN samples. The proofs are provided in Secs. S1.2.2 and S1.3.2, along with explicit perturbation bounds that show how the error depends on the dimension and on the conditioning of the covariance matrices.

Weakness 4 and Question 2: Applications

Our work is primarily theoretical: its main contribution is the exact solution of the Blackwell PID for Gaussian systems, together with efficient algorithms to compute it. For this reason, we did not include an analysis of real data.

At the same time, the paper does include an application to a model system, the optimal-control example of Sec. 7. This example shows how a controller combines information from its sensor and its memory. As the sensor noise grows, the controller passes from a regime where memory is redundant with the sensor, to a synergistic regime, and finally to a regime where the sensor is redundant with memory. These regimes cannot be identified from mutual information terms alone.

Regarding real data, the Gaussian BROJA PID has previously been used to analyze neural recordings (Venkatesh et al., 2023) and multimodal benchmarks (Zhao et al., 2025). Since the BROJA measures coincide with our union information and synergy, our results apply directly to such analyses.

Reviewer_fFB1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops an exact computational characterization of Blackwell redundancy and union information for multivariate Gaussian source–target pairs. It establishes Gaussian optimality, expresses the quantities as convex log-determinant programs in target space, and derives a generalized-eigenvalue closed form for two-source union information and hence BROJA synergy. It also provides dimension reductions and numerical solvers, compares redundancy measures on illustrative Gaussian examples, and demonstrates the approach through scalability benchmarks and an LQG controller with sensor and memory inputs.

Critical strengths
  1. The theoretical contribution is substantive beyond the established Blackwell framework. Gaussian optimality for redundancy, the arbitrary-source target-space formulation, and the two-source union closed form provide useful advances over existing iterative Gaussian PID methods. In particular, optimizing in target space can substantially reduce the computational burden when the sources are high-dimensional.

  2. The paper connects its theoretical characterization to practical computation through dimension reductions, certified optimization, and scalability benchmarks. The ellipsoid interpretation makes the distinction between redundancy and union information intuitive, while the LQG example illustrates how the framework characterizes information shared between sensor measurements and memory.

Critical weaknesses

The statistical implications of exact subspace intersections are insufficiently discussed. Theorem 2 and Section 6 imply that redundancy can be discontinuous even when the full joint Gaussian covariance is nonsingular. For independent standard Gaussian Y1, Y2, N1, and N2, consider Y=(Y1,Y2), X1=Y1+N1, and X2=cos(theta)*Y1+sin(theta)Y2+N2. At theta=0, redundancy is 0.5log(2); for arbitrarily small nonzero theta, the observed one-dimensional subspaces have zero intersection and redundancy is zero. Consequently, applying the exact functional to unconstrained sample covariances can return zero even for two noisy observations of the same target feature, because the estimated directions are almost surely not exactly aligned. This does not contradict the population-level theorems, and discontinuity of Blackwell redundancy is known in prior work. However, its consequences for the proposed Gaussian computation materially limit the demonstrated applicability to empirical representation analysis. The known-covariance experiments do not address this limitation, and numerical optimality certificates should be distinguished from statistical accuracy.

Questions for authors

The following clarification requests concern the practical scope of the results:

  1. Does the perturbation example above correctly characterize the proposed redundancy? How does the implementation handle approximate subspace intersections and numerical rank thresholds? In particular, does the reported optimality certificate apply to the original covariance problem or to a thresholded approximation? Please clarify the implications for using sample covariance estimates.
  2. In Figure 2(d), two generic 64-dimensional source subspaces have zero intersection once the target dimension reaches 128. Can the authors distinguish timings for these zero-redundancy cases from cases requiring nontrivial optimization, and clarify how this affects the interpretation of redundancy scalability?
1 comment

We thank the referee for the careful reading of our manuscript. Both questions led to improvements, which we describe below.

Question 1: Discontinuity and estimation from sample covariances

The referee's example is correct: Blackwell redundancy can be discontinuous, even when the joint covariance is nonsingular.

As the referee notes, this discontinuity is known from prior work, and in our view it makes sense given the operational meaning of the measure (Kolchinsky, 2022, Secs. 5.5–5.6). Blackwell redundancy is the most information that one can have about the target without being able to outperform any single source on any decision problem. In the referee's example, for any θ≠0\theta\neq0 the two sources provide information about different directions of the target. Any informative channel would then outperform at least one of the sources on some decision problem, however small θ\theta is, and so redundancy vanishes.

At the same time, we agree that the previous version did not discuss the consequences for estimation. To address this, we now show that redundancy is guaranteed to be continuous under a simple condition on the ranks: it is a locally Lipschitz function of the covariance matrices whenever all sources except possibly one have full rank, rank⁡ΣYXi=dY\operatorname{rank}\Sigma_{YX_i}=d_Y (Proposition 1 in Sec. 5.1). More generally, we show that redundancy is locally Lipschitz at almost all covariance matrices, for any source and target dimensions. Union information and synergy are continuous without any rank condition (Proposition 2 in Sec. 5.2). Wherever these quantities are locally Lipschitz, the PID can be consistently estimated from data, with error of order N−1/2N^{-1/2} for NN samples. The proofs are provided in Secs. S1.2.2 and S1.3.2, along with explicit perturbation bounds.

A discontinuity therefore requires two or more sources that each provide information about only some directions of the target and that are exactly aligned, as in the referee's example at θ=0\theta=0. Such cases form a set of covariance matrices of measure zero.

Regarding the implementation, the numerical rank thresholds account only for roundoff error. The optimality certificate bounds the optimization error for the covariance matrices provided as input, not the statistical error (see also the new "Numerical rank" paragraph in Sec. S5.2).

Question 2: Target-dimension benchmark

The referee is correct, and we thank them for noticing this. In the previous Fig. 2(d), the two source subspaces generically do not intersect once dY≥128d_Y\ge128, so the redundancy timings in that range did not involve any nontrivial optimization.

We replaced panels (d,i) of Fig. 2, and the corresponding panels of Fig. S1, with a new experiment in which the sources are high-dimensional, d1=d2=1000d_1=d_2=1000, and the target dimension is varied up to dY=1000d_Y=1000 (Sec. S5.5). Each source then observes every target direction, so redundancy is strictly positive and its optimization is carried out over full dY×dYd_Y\times d_Y matrices.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Summary of revisions

We thank the referees for their careful reading and useful comments. The main changes in the revised manuscript are:

  1. New results on continuity and estimation from data (Propositions 1 and 2 in Secs. 5.1 and 5.2, with proofs in Secs. S1.2.2 and S1.3.2).
  2. A new paragraph in the Introduction, which summarizes our contributions and their relation to prior work.
  3. A new statement of Theorem 4, which is symmetric in the two sources and allows singular covariances.
  4. A new benchmark in Fig. 2(d,i), where the redundancy optimization is carried out in the full target space.
  5. A new Sec. S2 on rate-constrained Blackwell redundancy, which connects Blackwell redundancy with the Gaussian information bottleneck, and a corresponding paragraph in Sec. 5.1.

We have also polished the text and simplified some of the derivations. In addition, we fixed some minor issues in our numerical implementation and reran the numerical experiments. Substantive changes in the manuscript are marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.