- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a unifying theoretical framework that maximizes shared information between two views while limiting what each representation retains about its own view. Each view has its own compression cost, and the authors demonstrate that, for Gaussian data, while the transition to new active modes is smooth when one of the two views is unconstrained (i.e., the traditional IB settings), there is a clear switch in the transition (i.e., a given mode is suddenly on or off) when both views are constrained. The proposed framework unifies methods such as CCA, Imax, and symmetric IB, and offers a qualitative interpretation of several self-supervised learning methods.
Critical strengths
- The paper proposes a clear, unifying framework for various methods. This is a great tool for better understanding the inner workings of various models and their failure modes (e.g., a trivial encoder is always a local optimum, which may help to explain collapse issues in SSL).
- The mode-switch part with continuous IB at the end of the line is very interesting and novel. If this holds beyond Gaussian settings, it has various implications for the community, for example:
- a grid search must be sufficiently fine-grained so as not to miss a switch
- unequal views need unequal compression
- strongest shared features are learned first, and the number of features learned depends on the compression applied to both views
Critical weaknesses
- Some related works are missed or incorrectly reported. DVSIB (Nemenman and Martini, JMLR 2025) is never cited but optimizes the same symmetric compression objective (Eq. 14 of that paper, DVSIB no-recon version) and discusses links to methods such as Barlow Twins and deep CCA. It would be helpful to explain how the current findings relate to that paper. Unless I am mistaken, Dikshtein et al. (2022) already have separate compression terms for each view ( and from Eq. 7), but in the paper their work is reported as symmetric compression in several places (l. 389–391 and l. 430–431).
- There are no experiments at all. I know that this is a theoretical paper and that space is tight, but I would expect at least a synthetic experiment to show that the findings hold in practice.
Questions for authors
- I strongly encourage the authors to check that the findings about Barlow Twins and other common methods discussed in Nemenman et al. (2025) are consistent with yours.
- Could you clarify the discrepancy I reported for Dikshtein et al. (2022)? Is this a genuine mistake on my side? If not, could the authors correct this in the paper?
- It would be great to have at least a small synthetic experiment in the appendix (e.g., using jointly Gaussian views with known correlations) to show that the paper's claims hold in practice.
The author is thanked for the positive comments and constructive feedback. All the points raised by the reviewer are correct and have been acted on in the substantially enhanced manuscript.
First, we like to point out that the critical strengths raised by the author are excellent points which have now been noted in the revised submission.
Q1 DVSIB (Abdelaleem, Nemenman & Martini, JMLR 2025). We now cite and discuss it. Its no-reconstruction loss (their Eq. 14), IE(X;ZX)+IE(Y;ZY)−βID(ZX;ZY), is the variational counterpart of our functional on the diagonal θx=θy=1/β. The findings are consistent: DVSIB relates Barlow Twins to its deterministic, β→∞ limit with jointly Gaussian latents, where the loss reduces to −I(ZX;ZY)=21lndet(I−CC⊤). This is our θ→0 corner, where we place Barlow Twins and VICReg and where the solution is CCA. As the DVSIB paper also notes, the Barlow Twins shares one encoder across views, and we add this qualification. DVSIB's observation that compression is governed by latent dimension in the deterministic limit matches our rank result: at the origin, k latent dimensions retain the top k canonical modes. What our analysis adds for DVSIB-type objectives: a second temperature; an exact solution; and the prediction that along the single-β line modes switch on discontinuously, with a locally optimal solution at every β.
Q2 The reviewer is correct about Dikshtein et al. (2022) not being restricted to the symmetric ray and this is corrected in the text. The double-sided IB maximizes I(U;V) subject to separate constraints I(X;U)≤Cu and I(Y;V)≤Cv. Our functional is the Lagrangian version of that problem, with θx,θy the multipliers of the two rate constraints. However the rate constraint and Lagrangian solutions may not necessarily be equivalent when there is a gap duality, as noticed before for the Information Bottleneck. The text is augmented to discuss this. The Lagrangian formulation is the more appropriate form for learning since it provides a single functional to be optimized. The work of Dikshtein does not reveal the rich phase structure, treat the multivariate Gaussian case, or consider the Lagrangian form with two free multipliers. More technically, we show (i) the Lagrangian phase structure. The Gaussian-channel rate surface is bilinear, hence non-concave, near the origin, so the Lagrangian optimum must jump over a range of rates. That is the first-order transition, with the spinodal and coexistence conditions in closed form (Eqs. 20, 22) and the IB critical point as the endpoint of the transition line; (ii) the multivariate Gaussian solution, with proof that CCA-aligned encoders are optimal among linear-Gaussian encoders and that modes are admitted in order of ρi; (iii) the monotone alternating iteration in Lagrangian form, without the per-step constraint search.
Q3. Despite this being a theory paper, we agree with the reviewer that a synthetic experiment would be valuable to demonstrate validation of the claims. To this end, we carried out a numerical validation of the iterative algorithm of Section 2.3 against the theoretical closed form (new appendix). A new appendix is added to report the experiment with the setting suggested by the reviewer: jointly Gaussian views with known correlation. A bivariate Gaussian is tabulated on a 48×48 grid and the iterative algorithm of Section 2.3 (Eqs. 4–5) is run on it with unconstrained encoder tables, started at random and annealed upwards in temperature. Nothing Gaussian is assumed about the encoders.
For ρ=0.9,0.7,0.5 the synoptic information found by the algorithm matches the closed form to within 0.005 nats along the symmetric ray. On the asymmetric ray θy=4θx the two compression rates differ and each follows its predicted value ("unequal views need unequal compression"). The non-trivial solution persists beyond the temperature at which it stops being optimal and is lost abruptly at the predicted spinodal. This is the hysteresis expected of a first-order transition, and it supports the reviewer's point that a temperature grid must be fine and that a single run can sit on the wrong branch. Comparing F with that of the trivial encoder recovers the predicted switch. As the temperature is raised, weaker correlations are lost first, at the predicted temperatures ("strongest shared features are learned first"). The test is one mode at a time, which suffices because the modes decouple.