acceptodds
Under review as a conference paper at ICLR 2027

Synoptic Information: A Variational Principle for Joint Representation Learning

Abstract

How should two high-dimensional noisy views of the same system be jointly compressed so that their representations capture structure common to both while flexibly discarding view-specific detail? Learning such a joint reduced representation by exploiting shared internal structure can extract interesting features from the input data without the need for an external teacher. Here, an information-theoretic description of the problem is introduced and formalized as a two-temperature variational principle for the , the optimal number of bits shared between the two compressed views that maximally retain what each representation predicts about the other. Functional optimization yields exact coupled self-consistent equations for both encoders, readily solved by a generalized iterative algorithm. For multivariate Gaussian inputs, the synoptic information decomposes as a sum over canonical correlation modes, admitting closed-form analytical solutions. The phase diagram of optimal solutions in the temperature plane reveals phase transitions through nested regions of active modes and interpolates between differing representation learning frameworks, such as canonical correlation analyses and the information bottleneck, which appear as limiting cases.

Primary area:
learning theory
Keywords:
representation learning, information theory, self-supervised learning, compression

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_MXprOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a unifying theoretical framework that maximizes shared information between two views while limiting what each representation retains about its own view. Each view has its own compression cost, and the authors demonstrate that, for Gaussian data, while the transition to new active modes is smooth when one of the two views is unconstrained (i.e., the traditional IB settings), there is a clear switch in the transition (i.e., a given mode is suddenly on or off) when both views are constrained. The proposed framework unifies methods such as CCA, Imax, and symmetric IB, and offers a qualitative interpretation of several self-supervised learning methods.

Critical strengths
  • The paper proposes a clear, unifying framework for various methods. This is a great tool for better understanding the inner workings of various models and their failure modes (e.g., a trivial encoder is always a local optimum, which may help to explain collapse issues in SSL).
  • The mode-switch part with continuous IB at the end of the line is very interesting and novel. If this holds beyond Gaussian settings, it has various implications for the community, for example:
    • a grid search must be sufficiently fine-grained so as not to miss a switch
    • unequal views need unequal compression
    • strongest shared features are learned first, and the number of features learned depends on the compression applied to both views
Critical weaknesses
  • Some related works are missed or incorrectly reported. DVSIB (Nemenman and Martini, JMLR 2025) is never cited but optimizes the same symmetric compression objective (Eq. 14 of that paper, DVSIB no-recon version) and discusses links to methods such as Barlow Twins and deep CCA. It would be helpful to explain how the current findings relate to that paper. Unless I am mistaken, Dikshtein et al. (2022) already have separate compression terms for each view (CuC_u and CvC_v from Eq. 7), but in the paper their work is reported as symmetric compression in several places (l. 389–391 and l. 430–431).
  • There are no experiments at all. I know that this is a theoretical paper and that space is tight, but I would expect at least a synthetic experiment to show that the findings hold in practice.
Questions for authors
  • I strongly encourage the authors to check that the findings about Barlow Twins and other common methods discussed in Nemenman et al. (2025) are consistent with yours.
  • Could you clarify the discrepancy I reported for Dikshtein et al. (2022)? Is this a genuine mistake on my side? If not, could the authors correct this in the paper?
  • It would be great to have at least a small synthetic experiment in the appendix (e.g., using jointly Gaussian views with known correlations) to show that the paper's claims hold in practice.
1 comment

The author is thanked for the positive comments and constructive feedback. All the points raised by the reviewer are correct and have been acted on in the substantially enhanced manuscript.

First, we like to point out that the critical strengths raised by the author are excellent points which have now been noted in the revised submission.

Q1\textbf{Q1} DVSIB (Abdelaleem, Nemenman & Martini, JMLR 2025). We now cite and discuss it. Its no-reconstruction loss (their Eq. 14), IE(X;ZX)+IE(Y;ZY)−βID(ZX;ZY)I^E(X;Z_X) + I^E(Y;Z_Y) - \beta I^D(Z_X;Z_Y), is the variational counterpart of our functional on the diagonal θx=θy=1/β\theta_x = \theta_y = 1/\beta. The findings are consistent: DVSIB relates Barlow Twins to its deterministic, β→∞\beta\to\infty limit with jointly Gaussian latents, where the loss reduces to −I(ZX;ZY)=12ln⁡det⁡(I−CC⊤)-I(Z_X;Z_Y) = \tfrac12\ln\det(I - CC^\top). This is our θ→0\theta\to0 corner, where we place Barlow Twins and VICReg and where the solution is CCA. As the DVSIB paper also notes, the Barlow Twins shares one encoder across views, and we add this qualification. DVSIB's observation that compression is governed by latent dimension in the deterministic limit matches our rank result: at the origin, kk latent dimensions retain the top kk canonical modes. What our analysis adds for DVSIB-type objectives: a second temperature; an exact solution; and the prediction that along the single-β\beta line modes switch on discontinuously, with a locally optimal solution at every β\beta.

Q2\textbf{Q2} The reviewer is correct about Dikshtein et al. (2022) not being restricted to the symmetric ray and this is corrected in the text. The double-sided IB maximizes I(U;V)I(U;V) subject to separate constraints I(X;U)≤CuI(X;U)\le C_u and I(Y;V)≤CvI(Y;V)\le C_v. Our functional is the Lagrangian version of that problem, with θx,θy\theta_x, \theta_y the multipliers of the two rate constraints. However the rate constraint and Lagrangian solutions may not necessarily be equivalent when there is a gap duality, as noticed before for the Information Bottleneck. The text is augmented to discuss this. The Lagrangian formulation is the more appropriate form for learning since it provides a single functional to be optimized. The work of Dikshtein does not reveal the rich phase structure, treat the multivariate Gaussian case, or consider the Lagrangian form with two free multipliers. More technically, we show (i) the Lagrangian phase structure. The Gaussian-channel rate surface is bilinear, hence non-concave, near the origin, so the Lagrangian optimum must jump over a range of rates. That is the first-order transition, with the spinodal and coexistence conditions in closed form (Eqs. 20, 22) and the IB critical point as the endpoint of the transition line; (ii) the multivariate Gaussian solution, with proof that CCA-aligned encoders are optimal among linear-Gaussian encoders and that modes are admitted in order of ρi\rho_i; (iii) the monotone alternating iteration in Lagrangian form, without the per-step constraint search.

Q3\textbf{Q3}. Despite this being a theory paper, we agree with the reviewer that a synthetic experiment would be valuable to demonstrate validation of the claims. To this end, we carried out a numerical validation of the iterative algorithm of Section 2.3 against the theoretical closed form (new appendix). A new appendix is added to report the experiment with the setting suggested by the reviewer: jointly Gaussian views with known correlation. A bivariate Gaussian is tabulated on a 48×4848\times48 grid and the iterative algorithm of Section 2.3 (Eqs. 4–5) is run on it with unconstrained encoder tables, started at random and annealed upwards in temperature. Nothing Gaussian is assumed about the encoders.

For ρ=0.9,0.7,0.5\rho = 0.9, 0.7, 0.5 the synoptic information found by the algorithm matches the closed form to within 0.0050.005 nats along the symmetric ray. On the asymmetric ray θy=4θx\theta_y = 4\theta_x the two compression rates differ and each follows its predicted value ("unequal views need unequal compression"). The non-trivial solution persists beyond the temperature at which it stops being optimal and is lost abruptly at the predicted spinodal. This is the hysteresis expected of a first-order transition, and it supports the reviewer's point that a temperature grid must be fine and that a single run can sit on the wrong branch. Comparing F\mathcal F with that of the trivial encoder recovers the predicted switch. As the temperature is raised, weaker correlations are lost first, at the predicted temperatures ("strongest shared features are learned first"). The test is one mode at a time, which suffices because the modes decouple.

Reviewer_44sSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a multi-view information bottleneck framework that aims to maximize shared information between representations. The central idea is to restrict general linear Gaussian encoders to independent CCA-aligned channels and derive mode-wise optimal solutions, aiming to characterize the global optimum and phase transitions of the general linear Gaussian problem.

Overall: An interesting and interpretable theoretical analysis, but missing justification for CCA alignment, limited practical validation, and an insufficiently supported algorithmic component weaken its contribution. Borderline, leaning accept, but open to reassessment after rebuttal.

Critical strengths

1. Analytical tractability. The paper provides closed-form solutions for CCA-aligned Gaussian encoders, offering an interpretable characterization of the trade-off between shared information and compression.

2. Phase-transition analysis. The paper characterizes first-order phase transitions and discontinuous mode activation under two-sided compression, providing insights beyond classical Gaussian information bottleneck analysis.

Critical weaknesses

1. Missing optimality justification for CCA alignment. The analysis assumes independent encoding along CCA directions without establishing that this restriction does not compromise optimality among general linear Gaussian encoders. It is therefore unclear whether the derived mode-wise solutions also achieve the global optimum of the original problem.

2. Limited empirical validation. The evaluation is limited to visualizations of analytical results, without experiments demonstrating how the theory guides practical choices. This leaves its practical relevance and applicability beyond the analyzed Gaussian setting unclear. A focused learning task or mechanism experiment would help address this gap.

3. Unclear role and guarantees of the BA algorithm. The main Gaussian results follow from closed-form optimization, leaving the BA algorithm’s contribution unclear. The claimed monotonic improvement is not sufficiently justified for the proposed alternating updates, and the effectiveness of the annealing strategy in avoiding trivial or suboptimal solutions remains unverified.

Questions for authors

Please refer to weakness section.

1 comment

The reviewer is thanked for the positive comments and constructive feedback. We agree with, and address, all the comments and questions raised by the reviewer, and have incorporated all of them into the substantially revised manuscript.

W1. Optimality of CCA alignment\textbf{W1. Optimality of CCA alignment}

This is an important technical point and the new Appendix section indeed proves that every stationary point of F\mathcal F over arbitrary linear-Gaussian encoders (A,B)(A,B) is CCA-aligned, so the restriction costs nothing. In brief: the stationarity condition for each encoder forces it to share principal axes with what the other encoder has resolved (the Gaussian IB condition, applied to both sides). Combining the two conditions eliminates one encoder and leaves an equation for the other whose only remaining matrix is the diagonal matrix of canonical correlations, so both encoders must be diagonal in the canonical basis. Since the maximum of F\mathcal F is attained at a stationary point, the mode-wise solutions of Section 3.4 are the global optimum of the original matrix problem. The scope is linear-Gaussian encoders; among arbitrary encoders Gaussian channels are not optimal in general for the two-sided problem (Dikshtein et al., 2022).

W2. Empirical validation\textbf{W2. Empirical validation}

We agree with the author on the added value of providing experimental validation data. To this end, we carried out a numerical validation of the iterative algorithm of Section 2.3 against the theoretical closed form (new appendix). A bivariate Gaussian with correlation ρ\rho is tabulated on a 48×4848\times48 grid, and the updates of Eqs. 4–5 are run on it with unconstrained encoder tables (40 output symbols), started at random and annealed upwards in temperature. Nothing Gaussian is assumed about the encoders.

  • For ρ=0.9,0.7,0.5\rho = 0.9, 0.7, 0.5 on the symmetric ray, the synoptic information found by the algorithm matches the closed form to within 0.0050.005 nats and F\mathcal F to within 0.0030.003 nats.
  • Unequal compression. On the asymmetric ray θy=4θx\theta_y = 4\theta_x (ρ=0.8\rho = 0.8), the two rates I(SX;X)I(S_X;X) and I(SY;Y)I(S_Y;Y) differ and each follows its predicted value.
  • First-order transitions. In every run the non-trivial solution persists, with F\mathcal F negative, beyond the temperature at which it stops being optimal, and is lost at the predicted spinodal to within the temperature spacing. Comparing its F\mathcal F with that of the trivial encoder recovers the global optimum and the predicted coexistence temperature.

As the reviewer suggested, the experimental validation demonstrates how the theory guides practical issues. i) A single run reports a branch, not necessarily the optimum; ii) Temperature sweeps show hysteresis; iii) unequal temperatures produce unequal rates. We state the limits: the test is one mode at a time, which suffices because the modes decouple.

W3. Role and guarantees of the BA algorithm.\textbf{W3. Role and guarantees of the BA algorithm.}

  • The algorithm is now employed in the empirical validation requested by the reviewer (new Appendix). The general discrete algorithm also serves to validate the Gaussian theory within discretization error.

  • Monotonicity. With the YY-encoder held fixed, the update of the XX-encoder (Eq. 4) is an information bottleneck update for XX with SYS_Y as the relevance variable, so by the argument of Tishby et al. (1999) it cannot decrease F\mathcal F; the same holds for the YY update. Since F≤I(SX;SY)≤I(X;Y)\mathcal F \le I(S_X;S_Y) \le I(X;Y), the value of F\mathcal F converges. Section 2.3 now states this. As a check, we ran 120,000 half-steps on 300 random joint distributions and found no decrease beyond rounding error.

  • We make clear in the revised manuscript that the convergence of iterative algorithm is to a stationary point, not necessarily the global maximum. The same limitation holds for the original information bottleneck algorithm. Annealing from low temperature tracks the nontrivial branch, but as the experiment above shows it follows that branch into the metastable region up to the spinodal. The global optimum is identified by comparing F\mathcal F across branches (including the trivial one). Section 2.3 is revised to state this.

  • Another salient aspect of the iterative algorithm that we have learned from the new empirical validation is that a grid search must be sufficiently fine-grained so as not to miss a first-order transition.

Reviewer_ba9cOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides an information-theoretic formalization of the common machine learning setting of learning a representation by encoding two views of the same data. Under Markov assumptions, the authors characterize the objective as a functional that maximizes the mutual information between the compressed representations (which they call synoptic information), while minimizing the temperature-weighted mutual information between each view and their respective representation. After providing the stationary conditions for this functional, they give exact solutions for the analytically tractable setting where the data are jointly Gaussian and each encoder is linear Gaussian. They do this by finding the canonical modes underlying canonical correlation analysis (CCA), and leveraging the resulting independence of the variates to decompose the synoptic functional into a sum of independent scalar functions. From here, they can provide the closed form solutions to a single mode in terms of the temperature parameters and the representational covariance matrix. The authors give exact conditions for when the encoder exhibits phase transitions between trivial, non-trivial, or meta-stable states, and show that the number of modes (and transitions between modes) depends on the temperature. Finally, the authors show that this framework generalizes several existing ones.

Critical strengths

Formalizing the problem of learning joint embeddings as an information-theoretic objective, and solving the linear-Gaussian case, are important theoretical contributions. I find the framework to be an elegant generalization of the Gaussian information bottleneck to the case of two encoders, and I appreciated how the authors used canonical variates to simplify the complex functional to a sum of independent functions. The paper is well written and easy to follow.

Critical weaknesses

There are a couple of slight weaknesses. Section 3.4 after Eq. 21 was fairly hard to parse. I would appreciate if the authors made references to the phase diagram in Fig. 1 or otherwise provided some additional intuition. For example, it may be useful to show a plot of the free energy functional ff against the synoptic information log⁡(1+u)\log(1+u) for fixed ρ\rho over different distances along the ray of fixed ratio of u/vu/v to show the metastable, active, and non-spinodal (no second hump) regimes.

The notation uiu_{i} and viv_{i} to indicate encoder channel gains was somewhat confusing because UU and VV were already used in the SVD decomposition of the cross-covariance matrix CC. Using a different letter than uu would make it clearer that it is unrelated to UU (and the same for vv and VV).

To motivate Eq. 10, the authors mention that each canonical variate can be treated separately by individual channels of the encoder, but I felt this was not well motivated (see Q1). Moreover, as far as I can tell there is no mention of how different values of kXk_{X} and kYk_{Y} affect this derivation except for the clause that the authors "ignore the leftover coordinates" (line 214-5) (see Q2). It would be good to be more explicit about the preconditions for Eq. 10.

Questions for authors
  1. The encoders are restricted to be rectangular diagonal matrices, but it is unclear whether this is provably the global optimum. If BB is aligned, Chechik et al.’s Gaussian IB result says the best AA is also aligned, and vice versa. So aligned pairs are fixed points of the alternating optimization. However, that does not necessarily rule out a better non-aligned pair. In other words, using this mapping holds under the assumption of alternating optimization where one mapping is held fixed while the other is optimized, but if these are jointly optimized (as is often the case for joint embedding architectures), can we still guarantee that the best fixed point is always an aligned pair of encoders? How can we justify restricting the choice of AA and BB to be diagonal scaling matrices and not general ones?
  2. The method assigns independent encoder channels for each active canonical mode, but the method permits kXk_{X} and kYk_{Y} to be arbitrary size. How does the closed-form solution extend to accomodate the case when min⁡(kX,kY)\min(k_{X}, k_{Y}) is smaller than the number of active canonical modes rr? Is truncation of rr to the strongest canonical modes provably optimal under these constraints, or are additional assumptions required?
2 comments

The reviewer is thanked for the positive comments and constructive feedback. All comments and questions by the reviewer have been addressed and incorporated into the revised manuscript.

Q1\bf{Q1}: Is the aligned pair provably the global optimum?

This is an important question and took some time to resolve mathematically, but the answer is yes, among linear-Gaussian encoders. Every stationary point of F\mathcal F over arbitrary matrices (A,B)(A,B) is aligned (new Appendix A).

Setting ∂F/∂A=0\partial\mathcal F/\partial A = 0 shows that the XX-encoder must share its principal axes with what the YY-encoder has resolved, and vice versa. This is the Chechik et al. result, cited by the reviewer, applied to both sides. Substituting one condition into the other eliminates the YY-encoder and leaves a single matrix equation for the XX-encoder. The only other matrix in it is R2\mathsf{R}^2, the diagonal matrix of squared canonical correlations, so its solutions must be diagonal in the canonical basis. The same then holds for YY. F\mathcal F is bounded above and the compression costs grow without bound, so the global maximum is a stationary point and is therefore aligned. Eq. 10 is thus a consequence of optimality, not an ansatz, and the text is rewritten to state this.

Q2\bf{Q2}. What if min⁡(kX,kY)\min(k_X,k_Y) is smaller than the number of active modes?

The same result answers this with no extra assumption. An encoder with kXk_X outputs can resolve at most kXk_X directions of its input, and likewise for YY, so a stationary point contains at most k=min⁡(kX,kY)k = \min(k_X,k_Y) canonical modes and its value is a sum of at most kk terms fi+f_i^+. Because fi+f_i^+ is nondecreasing in ρi\rho_i, the optimum is the kk strongest canonical modes, each at its unconstrained optimal SNR, dropping those that are inactive. Truncation to the strongest modes is therefore provably optimal. Section 3.3 now states the precondition for Eq. 10 explicitly: kXk_X and kYk_Y at least the number of active modes, and the truncated statement otherwise.

Weakness comments\textbf{Weakness comments}

  1. Section 3.4 after Eq. 21. We adopt the reviewer's suggestion. New Appendix C and Figure 4 plot the single-mode functional along the stationary ray against the transmitted information at five temperatures for ρ=0.7\rho = 0.7, showing the active, coexistence, metastable, spinodal and trivial-only regimes. Section 3.4 points to this figure, and Appendix C relates the regimes to the boundaries in the phase diagram (Figure 2).

  2. Notation. Agreed. The channel SNRs ui,viu_i, v_i are renamed (to μi,νi\mu_i, \nu_i) throughout the manuscript so they cannot be confused with the SVD factors U,VU, V.

  3. Motivation of Eq10. Covered by Q1 and Q2 above: Section 3.3 now motivates the aligned form by the optimality result and states the condition on kX,kYk_X, k_Y. We also replaced the informal argument for the nesting of the active regions with a direct one (Section 4.1): the compression costs do not depend on ρ\rho while the shared term increases with it, so any resolution that makes a weaker mode worthwhile makes a stronger mode at least as worthwhile. Text has been added to describe how differing values of kxk_x and kyk_y affect the derivation.

Thank you for the additional proof and for clarifying the points I found confusing. I also appreciate the change in notation as suggested. Despite the similarity to DVSIB (Nemenman and Martini, JMLR 2025) noted by reviewer MXpr, I believe the jointly Gaussian regime explored in this paper provides a useful setting that admits exact solutions and complements the numerical results of DVSIB. I therefore maintain my score of clear acceptance (4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.