- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper derives an analytical solution of synergy and redundancy decompositions for multivariate Gaussian processes.
Critical strengths
The paper is conceptually sharp, with Theorems 1-3 containing the main results. All mathematical details are left for the Supplementary material. But the result is very strong, and I can't find any flaws in the derivations. The results are elegant by their generality and in the PID field are very relevant, providing analytical results that were missing. The results are likely to find broad application, since Gaussian approximations are common in practice.
Critical weaknesses
I think the paper is very strong and I do not have much to object.
Questions for authors
1-I think it would be enlightening to comment on the connection with the Gaussian Information Bottleneck (Chechik 2005). At first, given that Kolchinsky 2024 reformulated the Blackwell redundancy in terms of an information bottleneck, my first instinct was to compare the derivations here presented with the Gaussian bottleneck of Chechik, and see if it was superseded by it. It turns out, I think, that it is the opposite, apart from defining Gaussian redundancy one could use the tools developed in this paper to formulate more general versions of the Chechik results. Of course I do not expect the authors to do that here, but some comments on the connection, even if in the supplementary material, would be helpful, if anything to avoid my first reaction in the direction that some of the results might had overlapped with existing Gaussian information-bottleneck results.
2- In theorem 4, , not only one
3-Theorem 4 would be easier to understand if writing: For , let and solve the generalize eigenvalue problem , ....
The condition is needed to guarantee that you can define such that by symmetry and can be defined as , such that , . The condition is needed so that no negative eigenvalues exist, or otherwise would not be PSD. Note that singularities have been allowed for Theorem 2 and 3, although in Theorem 4 it would lead to an infinite .
So, this formulation would be clearer, but still it remains asymmetric in the role of and . This is reflected in the conditions , , and in the different roles they play in , .
A better slightly more general presentation can be used with , requiring . Then with , , and for orthogonal $U^T A U=P = diag(p_1,\ldots,p_{d_Y})$. Then with , we obtain and , and .
Done in this way Eq 14 becomes symmetric in and . You get . Eq 14 becomes then more transparent, you have $I_{\cup} = -\frac12\log|C_1+C_2| -\frac12\sum_j \log\min(p_j,1-p_j)$, so it is interpretable that in the spectral domain in each mode the smaller posterior is selected. Apart from being more interpretable, it is slightly more general than requiring , or , .
4-Note that in fact, for , then happens if and only if , as can be seen from , which happens iff for both. Therefore, the symmetric formulation with can be related to the considerations in line 250 above Eq (9).
Response
We thank the referee for the positive assessment of our work and for the detailed and useful suggestions.
Question 1: Connection to the Gaussian information bottleneck
We thank the referee for this suggestion, and we agree with the referee's intuition: our results are not superseded by those of Chechik et al. (2005). The Gaussian information bottleneck (IB) concerns a single source and trades off prediction against compression. Blackwell redundancy concerns several sources and places no cost on compression. The bottleneck formulation of Blackwell redundancy in Kolchinsky (2024) is also different, since it relaxes the requirement that the channel be produced from every source rather than adding a cost on compression.
Following the referee's suggestion, we added a new Sec. S2 that connects the two problems. There we define a rate-constrained version of Blackwell redundancy: this reduces to Blackwell redundancy when the rates are unconstrained, and to the IB problem when there is a single source (it also encompasses the multi-view IB of Huang et al. (2022)). We analyze this generalized formulation by extending our proof of Theorem 2, using an approach similar to the analysis of the vector Gaussian CEO problem by Uğur et al. (2020), which we now cite. As the referee anticipated, this provides an alternative proof of Gaussian optimality for the IB and related objectives. We mention this connection in Sec. 5.1 and refer to Sec. S2 for details.
Questions 2–4: Presentation of Theorem 4
We thank the referee for these suggestions, which led us to restate Theorem 4. We agree that the previous statement treated the two sources asymmetrically, and that the notation for the optimizer was unclear.
The revised theorem considers any invertible matrix V that diagonalizes both conditional covariances, V⊤ΣY∣X1V=diag(a1,…,adY) and V⊤ΣY∣X2V=diag(b1,…,bdY). Union information is then achieved by a Gaussian channel with posterior covariance V−⊤diag(min(a1,b1),…,min(adY,bdY))V−1, now written with all of its diagonal entries, and it is given by
+\tfrac14\sum_{j}\left\lvert\ln\frac{a_j}{b_j}\right\rvert .$$ This statement is symmetric in the two sources. It also has the interpretation suggested by the referee: $a_j$ and $b_j$ are the posterior variances left by the two sources along common directions of the target, and the union channel keeps the smaller one in each direction. We now mention this interpretation after the theorem. The matrix $V$ is not unique, since rescaling its columns multiplies $a_j$ and $b_j$ by the same factor, which leaves the theorem unchanged. Different choices of scale recover the two earlier formulations. The referee's formulation corresponds to $V^\top(\Sigma_{Y|X_1}+\Sigma_{Y|X_2})V=I$, which gives $a_j=p_j$ and $b_j=1-p_j$. The formulation of the submitted version corresponds to $V^\top\Sigma_{Y|X_1}V=I$, which gives $a_j=1$ and $b_j=\lambda_j$. We state the theorem without fixing the scale, so that it contains both formulations. Regarding singular matrices, the revised theorem no longer assumes that the conditional covariances are positive definite: it states that $I_\cup=+\infty$ if any $a_j$ or $b_j$ vanishes. Theorem 3 was updated in the same way, so that Theorems 2–4 now all allow singular conditional covariances. The referee is also right that $C_1+C_2\succ0$ is equivalent to $\ker C_1\cap\ker C_2=\lbrace0\rbrace$, which is the condition for finite redundancy stated in Sec. 5.1.