acceptodds
Under review as a conference paper at ICLR 2027

Gaussian Joint Embeddings For Self-Supervised Representation Learning

Abstract

Self-supervised representation learning traditionally relies on deterministic predictions or contrastive objectives, which struggle with multi-modal ambiguities and often require heuristic asymmetries to prevent dimensional collapse. We propose a shift to generative joint modeling via Gaussian Mixture Joint Embeddings (GMJE). By explicitly modeling the joint density of context and target representations, GMJE replaces black-box prediction with closed-form conditional inference, natively providing uncertainty estimation and covariance-aware geometric regularization. We identify a fundamental optimization vulnerability in empirical batch modeling, the Mahalanobis Trace Trap, and resolve it through a suite of robust parametric, topological, and adaptive architectures. Further, we show that standard contrastive learning is a degenerate, non-parametric special case of GMJE, motivating our introduction of a dynamic Sequential Monte Carlo (SMC) memory bank to replace standard uniform FIFO queues. Empirical evaluations confirm that GMJE recovers complex multi-modal structures, achieves competitive discriminative performance, and constructs a continuous latent manifold capable of high-fidelity unconditional generative sampling.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Supervised Learning, Representation Learning, Gaussian Mixture Models, Generative Modeling, Contrastive Learning, Sequential Monte Carlo

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_k3dZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper reformulates self-supervised joint embedding learning not as a deterministic prediction or contrastive objective, but as a problem of modeling the joint probability density of context and target representations. It first proposes Gaussian Joint Embedding (GJE), which models p(zc,zt)p(z_c, z_t) as a joint Gaussian distribution, enabling closed-form conditional prediction and uncertainty estimation via Gaussian conditioning. To address the limitations of a unimodal Gaussian, specifically the tendency toward over-contraction, the paper then extends this to Gaussian Mixture Joint Embedding (GMJE), presenting several variants such as GMJE-Proto, GMJE-MDN, and GMJE-GNG.

This paper identifies and analyzes the "Mahalanobis Trace Trap," a phenomenon where the Mahalanobis data-fit term is effectively nullified, causing the gradient to vanish when the empirical batch covariance is used directly within a Gaussian negative log-likelihood (NLL) objective. Furthermore, under specific regularization, prototype approximation, and low-temperature assumptions, it interprets InfoNCE as the conditional likelihood of a non-parametric, isotropic GMJE. Building on this, it proposes SMC-GMJE, which replaces the conventional FIFO memory bank with a Sequential Monte Carlo (SMC)-based particle bank.

The experiments are threefold. In synthetic multi-valued regression tasks, mixture-based variants demonstrate a superior ability to recover multimodal conditional structures compared to deterministic or unimodal predictors. On CIFAR-10, the SMC memory bank is compared against a MoCo-style FIFO queue; it is noted that, given a sufficient memory budget, the SMC-based approach consistently outperforms the FIFO-based counterpart. Finally, the generative capacity of GMJE by sampling from the learned latent density and reconstructing images through an associated decoder is evaluated. The paper makes an ambitious attempt to bridge predictive self-supervised learning (SSL), contrastive learning, and density estimation within a single framework—specifically through the lens of probabilistic joint modeling. However, the current version lacks empirical evidence to support this broad conceptual framework; most notably, the parametric GMJE, a core component of the paper’s approach, proves to be a failure when tested on standard representation-learning benchmarks, which represents a fundamental issue.

Critical strengths
  1. An interesting probabilistic perspective. The approach of modeling p(zc,zt)p(z_c, z_t) directly by factoring it as p(zt∣zc)p(zc)p(z_t|z_c)p(z_c) to unify prediction and latent-space regularization within a single likelihood framework is compelling. In particular, the motivation to address the tendency of deterministic latent prediction to converge toward the conditional mean (in cases of multi-modal conditional distributions) by employing mixture modeling is clear.

  2. The analysis of the "Mahalanobis Trace Trap" is lucid. The observation that the Mahalanobis term cancels out to a constant, d/2d/2, when the empirical covariance C=N−1Z⊤ZC=N^{-1}Z^\top Z is used in the Gaussian negative log-likelihood (NLL) for the same set of samples is both concise and useful. It explicitly demonstrates why a naive covariance-based representation objective can lead to collapse. However, as noted later, the degree of novelty regarding this algebraic observation itself warrants further justification.

  3. The synthetic experiment effectively visualizes the motivation. In a multi-valued target setting, the fact that a deterministic MSE-based predictor averages across modes, whereas mixture models (such as GMJE-MDN) successfully recover distinct conditional branches, is intuitive and serves as an effective illustration of the paper’s core motivation.

  4. The results using SMC in a memory-constrained setting are intriguing. In experiments on CIFAR-10 with a small memory bank (M=256M=256) over 1,000 epochs, the SMC-based bank achieves a linear-probe accuracy of 81.99%, compared to 78.08% for a FIFO-based approach. This finding suggests that adaptive memory management is a promising area for future research.

Critical weaknesses
  1. There is a significant gap between the paper’s central methodological contribution and its actual empirical validation. This is the most critical issue. The paper’s title and core framing center on "Gaussian Mixture Joint Embeddings" (GMJE), presenting GMJE-Proto, GMJE-MDN, and GMJE-GNG as primary instantiations of joint-density-based self-supervised learning (SSL). However, these parametric GMJE variants are not properly evaluated on the standard CIFAR-10 representation-learning benchmark. The authors state that they did not report results for these variants because they proved difficult to optimize when trained from scratch. Instead, the primary vision-related results are derived from "SMC-GMJE," which retains the architecture, augmentations, and EMA targets of MoCo v2, replacing only the FIFO memory update mechanism with SMC. While this may support the utility of an SMC-based memory bank, it does not validate the effectiveness of the paper’s central claim: parametric probabilistic joint-density representation learning.

  2. The methodological scope is overly broad, and the coherence among the various contributions is not sufficiently established. The paper attempts to encompass a wide array of elements—including GJE/GP regression, primal Gaussian modeling, GMMs, MDNs, Growing Neural Gas, an InfoNCE reinterpretation, SMC memory banks, and unconditional latent generation—under a single "GMJE" framework. In reality, however, these are distinct approaches at the algorithmic level. In particular, the relationship between parametric GMJE and SMC-GMJE is more of a conceptual reinterpretation than a set of different implementations of the same practical learning algorithm. Consequently, it remains unclear what the paper’s primary proposed method is, or which experiments are intended to validate it. The current structure bundles several potentially interesting ideas into one framework without providing adequate validation for each.

  3. The proposed method lacks competitive performance on standard SSL benchmarks. In the same 200-epoch CIFAR-10 setting, the linear-probe accuracy is 81.50% for SimCLR, 79.04% for BYOL, and 79.02% for MoCo v2, whereas it is 74.52% for SMC-GMJE. Consequently, the claim of "competitive discriminative performance" made in the abstract lacks convincing support based on the current evidence. While SMC outperforms FIFO in a specific memory-constrained setting (M=256M=256, 1000 epochs), this alone is insufficient to claim general competitiveness in self-supervised learning (SSL). A systematic comparison across a wider range of memory sizes, training budgets, and datasets—as well as against stronger baselines involving memory mechanisms or hard-negative mining—is required.

  4. The evidential value of the synthetic multi-modality experiment is limited. This experiment merely demonstrates well-known phenomena: that deterministic squared-error regression learns the conditional mean, and that MDNs/GMMs can represent multi-modal conditional densities. In fact, the MDN itself is a classical method specifically designed to address such multi-valued regression problems. Therefore, while this experiment serves as a useful illustration of the motivation, it does not validate the claim that GMJE discovers meaningful multi-modal semantics within high-dimensional learned representations. Experiments that directly evaluate multi-modality or calibrated predictive uncertainty in a realistic SSL setting are necessary.

  5. The framing regarding an "exact" correspondence between InfoNCE and GMJE is overly strong. The correspondence presented in the proposition relies on several assumptions, including L2 normalization, the approximation between stored context prototypes and paired target keys, and a low-temperature limit. The authors themselves acknowledge these conditions in the "limitations" section. Thus, characterizing InfoNCE simply as "exactly a degenerate special case of GMJE" conveys a stronger claim of equivalence than is actually the case for InfoNCE in general. It is necessary to clearly distinguish the specific conditions under which equality holds from the aspects that rely on approximations or limiting arguments. 6. The evaluation of the generative modeling claim is insufficient. In the MNIST experiment, the authors claim that latent samples from the GMJE (Generative Modeling Joint Embedding) framework produce qualitatively superior decoded digits; however, this assessment relies heavily on t-SNE, sample visualization, and covariance/prototype statistics. Furthermore, a separate decoder must be trained to perform image generation. Consequently, the evidence is inadequate to support the strong claim of "high-fidelity unconditional generative sampling." At a minimum, the study requires quantitative metrics for sample quality and diversity, comparisons against stronger latent-density baselines, and results across multiple random seeds.

  6. The experimental robustness and scale are insufficient relative to the paper's central claims. The practical vision evaluation is limited to the CIFAR-10/ResNet-18 scale, while the generative experiments are confined to the MNIST dataset. The authors themselves acknowledge that scaling to larger backbones and datasets remains a limitation or a topic for future work. Given the paper's scope, i.e., proposing a new, general self-supervised learning (SSL) paradigm, the current evidence is best characterized as a proof-of-concept.

Questions for authors
  1. What exactly is the primary proposed method in this paper? Given that GMJE-Proto/MDN/GNG and SMC-GMJE are algorithmically quite distinct, please clarify what concrete algorithmic benefit, beyond a mere probabilistic interpretation, is gained by grouping them into a single family of methods.

  2. If the central contribution of the paper is parametric GMJE, why are the representation-learning results for parametric GMJE, GMJE-MDN, and GMJE-GNG on CIFAR-10 not provided? The current remark in the appendix, i.e., labeling them as "unstable or under-tuned", fails to address the lack of empirical validation for the core method. Is there any evidence that these methods can be trained stably under standard semi-supervised learning (SSL) settings?

  3. In the 200-epoch benchmark, SMC-GMJE underperforms MoCo v2 by approximately 4.5 percentage points, whereas in the M=256M=256, 1000-epoch setting, it outperforms FIFO by 3.91 points. How can one disentangle whether this performance difference stems from the memory size, the training duration, or the SMC mechanism itself? Could you provide a sweep across memory size and training duration?

  4. Regarding the SMC memory bank, it seems appropriate to compare it not only against FIFO but also against existing methods such as hard-negative mining, prioritized sampling, and reservoir or importance sampling. What are the empirical or theoretical advantages of SMC compared to these approaches?

  5. What objective is obtained if the low-temperature limit and the proxy assumption zc(m)≈zt(m)z_c^{(m)} \approx z_t^{(m)} are removed from the InfoNCE equivalence derivation? Please clarify whether the standard finite-temperature InfoNCE is exactly equivalent to the GMJE likelihood, or if it can only be recovered under a specific limiting construction.

  6. The "Mahalanobis Trace Trap" appears to be an identity involving a covariance/Mahalanobis quadratic form that arises when C=N−1Z⊤ZC = N^{-1}Z^\top Z is substituted back into the expression using the same observations. What makes this observation itself a novel result compared to existing literature on Gaussian MLE/covariance estimation? If the claim is that the implications for representation learning are novel, please clarify that distinction.

  7. GMJE is presented as having uncertainty estimation as a key advantage. Then why was uncertainty calibration not quantitatively evaluated in the actual experiments? Could you provide evaluations such as NLL, calibration error, coverage, or OOD uncertainty?

  8. In the MNIST generation experiment, is it possible to quantitatively evaluate sample quality/diversity, beyond just visual quality? Furthermore, does the advantage of GMJE persist if a stronger post-hoc density estimator—with the same capacity and tuning budget—is applied to the SimCLR latents?

1 comment

Thank you for the detailed assessment. due to word limit, we address the 8 questions and criticisms below.

  1. method and unity. The central formulation is joint density learning p(zc,zt)=p(zt∣zc)p(zc)p(z_c,z_t)=p(z_t\mid z_c)p(z_c) (Eq. 1), giving conditional prediction and an explicit context marginal. Parametric GMJE (Proto/MDN/GNG) models this with global, context-dependent, or adaptively grown mixture components (Secs. 4, C). SMC-GMJE instead uses the non-parametric contrastive/memory-bank limit (Secs. 4.1-4.2). They are distinct algorithms: grouping them identifies which assumptions yield conditional routing versus instance discrimination. We will clarify this division.

  2. Missing parametric CIFAR-10 results. These variants were attempted but were unstable/under-tuned in that 200-epoch from-scratch regime (Sec. 5.2; Appendix O.3); we have no defensible CIFAR-10 accuracy or stability claim for them. EM/GNG/MDN work on controlled conditional tasks (Figs. 2-3, 8-9), and parametric GMJE is trained for MNIST sampling (Sec. 5.3). We accept that full-image discriminative SSL remains unvalidated for these variants.

  3. Memory vs. duration. The +3.91-point comparison fixes M=256M=256, backbone and 1000 epochs, changing only FIFO to SMC (78.08% to 81.99%, Fig. 4); it therefore isolates the bank-update choice within that setting. The 200-epoch, M=4096M=4096 benchmark (Table 1) gives 74.52% SMC vs. 79.02% MoCo v2, and cannot be combined with the former to infer an across-budget trend. No size/schedule sweep is reported; we claim a narrow-budget gain.

  4. SMC vs. hard-negative/prioritized/reservoir sampling. SMC weights keys by current query affinity, then resamples a fixed-capacity bank (Eq. 11; Algorithm 1), retaining selected older keys beyond FIFO expiry. Its observed benefit over FIFO is accompanied by falling effective sample size (Fig. 4). Unlike uniform reservoir sampling, selection is data-dependent; it maintains a persistent resampled population. We have not shown superiority to those alternative methods. Multinomial resampling is a stochastic approximation to weighted expectations, not an exact finite-bank equality; duplication can reduce diversity.

  5. InfoNCE without the limiting assumptions. For the non-parametric KDE, the general conditional remains p(zt∣zc)=∑mrm(zc) N(zt;zt(m),τI)p(z_t\mid z_c)=\sum_m r_m(z_c)\,\mathcal N(z_t;z_t^{(m)},\tau I), where rm(zc)∝πmN(zc;zc(m),τI)r_m(z_c)\propto\pi_m\mathcal N(z_c;z_c^{(m)},\tau I). With normalized embeddings and the proxy zc(m)=zt(m)z_c^{(m)}=z_t^{(m)}, the posterior over component indices has a finite-temperature softmax form. But the continuous target-density NLL also includes the target-mixture terms; identifying it with InfoNCE requires the matching-component dominance/low-temperature approximation in Proposition 1. Thus standard finite-temperature InfoNCE is not generally identical to arbitrary GMJE conditional NLL; we will qualify the word "exact".

  6. Mahalanobis Trace Trap and collapse. We agree that 12N∑izi⊤C−1zi=d/2\frac{1}{2N}\sum_i z_i^\top C^{-1}z_i=d/2 for C=Z⊤Z/NC=Z^\top Z/N is a standard covariance identity (Eq. 5), not a new algebraic theorem. Its contribution here is identifying why the current-batch Gaussian NLL loses its Mahalanobis data-fit gradients when the encoders and covariance are optimized through the same samples. Independent/lagged covariance or conditional modeling avoids this specific cancellation; it does not establish collapse-free learning. Appendix N.4 explicitly warns that a separately learned covariance and encoder can still shrink together (the log-determinant cheat). We will distinguish these failure modes and qualify broad anti-collapse claims.

  7. Uncertainty evidence. GMJE provides conditional variances and mixture weights analytically (Eq. 7). In synthetic tasks with known noise 0.050.05 and equal branch probabilities, the learned MDN standard deviations and weights are visually close to these values (Figs. 3, 9). This tests basic recovery, not quantitative calibration on real images; coverage, held-out conditional NLL and OOD calibration were not measured.

  8. MNIST quality, matched baselines, and scale. Sec. 5.3 compares a learned GMJE 50-component density with a post-hoc 50-component GMM on SimCLR features and a unimodal GJE; Fig. 5 and Table 2 provide qualitative samples and latent-geometry diagnostics (prototype spread 1.43 vs. 0.55 for the SimCLR post-hoc GMM). Each representation uses a separate decoder. This demonstrates latent sampling, not high-fidelity image generation. A capacity/tuning-matched post-hoc estimator, FID-like quality/diversity metrics, independently trained seeds, and larger datasets are not supplied. We are working on this and expect to have an updated version.

  9. Scope/framing. The experiments establish a conditional-modeling proof of concept and a controlled memory-bank gain, not a broadly competitive SSL system. We will separate parametric GMJE from SMC-GMJE in the presentation and narrow the abstract's claims of "competitive" representation learning and "high-fidelity" generation.

Reviewer_em2qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce GMJE (and GJE), a family of self-supervised learning methods based on Gaussian joint embeddings. The method builds upon JEPA by replacing the learned predictor with a Gaussian (mixture) model of the joint context–target embedding density, from which predictions follow by conditioning; a dual variant uses a Gaussian process. The authors also show, under approximations that InfoNCE is a degenerate case of GMJE. Finally, GMJE is validated on a synthetic dataset, representation learning on CIFAR-10, and unconditional generation on MNIST.

Critical strengths
  • Proposition 1 is elegant and connects InfoNCE to Gaussian-mixture density estimation.
Critical weaknesses
  • The authors do not elaborate on the differences between GMJE and [1,2], which makes it hard to establish the significance of the proposed method. On a related note, the authors introduce several methods in the GMJE family, and it would be interesting for the readers to introduce an ablation on the different methods, or to move them to the appendix and spend more space addressing limitations, related works, a larger experimental campaign.

  • The experimental evidence is thin: the authors evaluate the quality of their representations only on CIFAR with a limited set of competitors which do not include standard baselines. Additionally, the authors do not test their representations on more realistic datasets, making it difficult to test whether the proposed method is actually useful.

[1] Yongchao Huang ; Vjepa: Variational joint embedding predictive architectures as probabilistic world models

[2] Moritz Gogl and Christopher Yau ; Var-JEPA: A Variational Formulation of the Joint-Embedding Predictive Architecture – Bridging Predictive and Generative Self-Supervised Learning

[3] Bardes et al. ; Vicreg: Variance-invariance-covariance regularization for self-supervised learning

[4] Assran et al. ; Self-supervised learning from images with a joint-embedding predictive architecture

Questions for authors
  • Unlike [1], which models only the conditional p(zt∣zc)p(z_t | z_c), GMJE also models the marginal p(zc)p(z_c). Can the authors discuss what this additional term brings in practice?
  • Can the authors provide empirical evidence that the proposed method works on more difficult datasets and add more recent and competitive baselines such as [3,4]? I understand that providing additional experiments at rebuttal time is difficult but the lack of experimental evidence is a big shortcoming of the paper, especially when the authors themselves list it as one of the contributions.

[1] Yongchao Huang ; Vjepa: Variational joint embedding predictive architectures as probabilistic world models

[2] Moritz Gogl and Christopher Yau ; Var-JEPA: A Variational Formulation of the Joint-Embedding Predictive Architecture – Bridging Predictive and Generative Self-Supervised Learning

[3] Bardes et al. ; Vicreg: Variance-invariance-covariance regularization for self-supervised learning

[4] Assran et al. ; Self-supervised learning from images with a joint-embedding predictive architecture

3 comments

Thank you for the constructive review and for recognizing the InfoNCE connection.

  1. Difference from VJEPA and Var-JEPA; role of p(zc)p(z_c). Whereas VJEPA models a future-latent conditional, GMJE explicitly models the joint embedding density p(zc,zt)=p(zt∣zc)p(zc)p(z_c,z_t)=p(z_t\mid z_c)p(z_c) (Eq. 1). This supplies a normalized context marginal alongside conditional prediction, yielding Gaussian/GMM conditional means, covariances and mixture responsibilities (Eqs. 4, 6-7). The term −log⁡p(zc)-\log p(z_c) couples representation learning to the global context-density geometry; in contrast, conditional fitting alone need not control this marginal. For a fixed or appropriately constrained covariance, its Mahalanobis and log-determinant terms provide a model-based geometry penalty. Unlike VJEPA's conditional latent prediction and Var-JEPA's generative ELBO for coupled latent variables, GMJE directly fits a normalized Gaussian-mixture density of the learned context-target embeddings; VICReg instead uses explicit variance/covariance penalties. Importantly, the marginal alone does not guarantee non-collapse when embeddings and covariances co-adapt (Appendix N.4). The experiments do not contain a clean p(zc)p(z_c)-on/off ablation, so its separate quantitative contribution is not established; we will distinguish the mathematical role from an empirically isolated gain.

  2. Why multiple GMJE variants? They address different limitations of one modeling formulation: a single Gaussian averages over modes; Gaussian process joint modelling replaces the JEPA predictor by a GP model; GMJE-Proto uses global learnable joint modes (Eq. 8); GMJE-MDN routes conditional mixtures using zcz_c alone (Eq. 9); GMJE-GNG adapts prototype topology; and SMC-GMJE is a non-parametric, contrastive/memory-bank branch, not an interchangeable optimization of the same parametric loss. The submitted synthetic comparison includes deterministic JEPA, Gaussian GJE, single- and three-component GMJE, GNG and MDN (Sec. 5.1; Figs. 2-3, 8-9). It isolates their conditional expressiveness, not their image-SSL performance. We will reorganize the methods, make a table for clear comparison, and make this separation explicit.

  3. Empirical evidence and missing competitive baselines. Table 1 does include SimCLR (81.50%), BYOL (79.04%), and MoCo v2 (79.02%), with SMC-GMJE at 74.52% in the 200-epoch CIFAR-10 setting. We do not claim it wins that comparison. The more informative controlled mechanism test holds the MoCo-style architecture and M=256M=256 bank fixed for 1000 epochs and changes the update rule: 81.99% SMC vs. 78.08% FIFO (Fig. 4). This is evidence for adaptive-bank utility in a constrained regime, not general superiority. VICReg and SwAV were attempted but not reported because their short-run optimization was unstable/under-tuned (Appendix O.3); we agree this is not a satisfactory competitive comparison. I-JEPA and larger/harder datasets are also not evaluated, and neither are CIFAR-10 parametric GMJE results. We are currently investigating these, and expect to give an updated version.

  4. Practical scope. MNIST (Sec. 5.3) demonstrates latent sampling with a learned 50-component parametric mixture and a separately trained decoder, but only provides qualitative sample comparisons and geometric diagnostics. We will narrow the abstract's competitiveness and generation language and clearly state which contribution each experiment validates. Broader image benchmarks, matched compute and explicit marginal-term ablations are needed to establish scale and relative performance - they will be supplied in an updated version.

  5. the references are very helpful - thanks for this. we shall have a careful look and update accordingly.

I thank the authors for their rebuttal. I agree with the authors about the proposed changes and directions to improve the paper, especially on the experimental side. However, at the current state, the paper still lacks the experimental depth necessary to support the work, and the presentation of some results and discussions should be reorganized, such as the discussion in appendix N.4, which could find space in the main part of the paper. Therefore, I will keep my score as is.

Thank you for the constructive follow-up and for recognizing the value of the revisions. We agree that stronger empirical comparisons are needed to establish the broader usefulness of parametric GMJE. We have made further revisions in the rebuttal version, following reviewers advice.

Presentation and collapse analysis. We have reorganized the methods and added Table 1 to distinguish Gaussian GJE, parametric GMJE-Proto/MDN/GNG, and the non-parametric SMC memory-bank branch. Following your suggestion, we have also moved the key scale-collapse counterexample from Appendix N.4 into Section 3, immediately after the Mahalanobis Trace Trap analysis. The main text now explicitly distinguishes empirical-covariance trace cancellation from co-adaptive embedding–covariance collapse, demonstrating why avoiding the former does not guarantee non-collapsed representations. Appendix N.4 has been condensed to discuss the corresponding remedies and their limitations. We hope these changes improve both the clarity and mathematical precision of the manuscript.

Experimental evidence. We have clarified that the synthetic MDN experiments demonstrate conditional multi-modality, the CIFAR-10 comparison isolates the SMC-versus-FIFO memory-bank update (81.99% versus 78.08% at M=256M=256 over 1,000 epochs), and the MNIST experiments illustrate qualitative latent sampling. These results support distinct aspects of the framework, although they do not yet establish competitive parametric GMJE representation learning. The revised manuscript also explicitly identifies the need for stronger SSL baselines, marginal-term ablations, repeated seeds, and quantitative generation comparisons.

We believe GMJE makes a distinctive, potentially first-of-its-kind contribution to JEPA-style representation learning. Rather than modeling only conditional target predictions or directly aligning embeddings, it learns an explicit joint context–target density, allowing prediction by probabilistic conditioning while also modeling the context marginal. This unifies analytic conditional inference, context-density geometry, and multi-modal prediction within a probabilistic joint-embedding framework. We believe this new modeling perspective, together with the theoretical analysis and initial empirical evidence, represents a meaningful contribution whose methodological value extends beyond the current experimental limitations.

Reviewer_Jo2BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes learning representations by fitting Gaussians (or a its mixture) to the embeddings of two views of the same image, instead of predicting one view from the other with squared error as in JEPA. It describes several ways to fit this mixture: learnable prototypes, a mixture-density network, growing neural gas, and a contrastive loss treated as a limiting case, with a particle-based memory bank. Experiments are reported mainly on a simple one-dimensional synthetic distribution.

Critical strengths

None.

Critical weaknesses

The proposed objective fails as a training criterion. A model that scores embeddings it produces itself can make the loss arbitrarily good by shrinking every embedding to a single point. The paperэs fixes either keep this score, so the same problem remains, or delete it, in which case nothing makes the two views of an image match. Besides, there are other serious problems in the claims and experiments.

The manuscript reads as AI-generated text (restated claims, paraphrasing, contradictions, etc.).

Questions for authors

See weaknesses.

3 comments

Thank you for your critical assessment. We address the central mathematical concern below.

  1. Does joint-density learning retain cross-view alignment? Yes. The objective is −log⁡p(zc,zt)=−log⁡p(zt∣zc)−log⁡p(zc)-\log p(z_c,z_t)=-\log p(z_t\mid z_c)-\log p(z_c) (Eq. 1). The conditional term is an explicit paired-view fitting signal, not merely a marginal volume penalty. GMJE-Proto fits the concatenated matched pair with shared mixture components (Eq. 8); GMJE-MDN retains −log⁡p(zt∣zc)-\log p(z_t\mid z_c) with mixture parameters depending only on zcz_c (Eq. 9); SMC-GMJE retains the positive-pair InfoNCE numerator (Eqs. 10--11). Thus removing a defective empirical-covariance quadratic does not, in these formulations, remove cross-view learning.

  2. Can learned embeddings and covariances collapse? you identify a real vulnerability of an unconstrained joint NLL: jointly shrinking embeddings, prototypes and covariance can make log⁡∣Σ∣→−∞\log|\Sigma|\to-\infty. Importantly, the submission explicitly identifies this separate log-determinant cheat (Appendix N.4). Decoupling covariance parameters from the current batch avoids the trace cancellation in Eq. 5, but does not by itself prove absence of all collapsed optima. Fixed/controlled covariance scales, normalized embeddings, or explicit dispersion constraints address different aspects of this issue; the contrastive SMC branch also inherits normalized embeddings and an explicit discrimination objective. We will correct any wording suggesting that a bare trainable joint likelihood universally prevents collapse. The distinction is between an analytically defective batch-covariance objective and practical, appropriately constrained instantiations - not a claim that likelihood alone suffices.

  3. What we actually demonstrated. The paper reports three different tests: (i) on two synthetic ambiguous-alignment tasks, GMJE-MDN recovers distinct conditional branches while deterministic MSE predicts their mean (Sec. 5.1; Figs. 2--3, 8--9); (ii) with a matched CIFAR-10 backbone, augmentations and M=256M=256 memory budget, replacing FIFO by SMC yields 81.99% vs. 78.08% after 1000 epochs (Sec. 5.2; Fig. 4); (iii) parametric GMJE is trained on MNIST and directly sampled through its learned mixture, with a separately trained decoder (Sec. 5.3; Fig. 5). These support specific conditional-modeling and memory-bank claims, not general state-of-the-art SSL or guaranteed anti-collapse. We explicitly report that the parametric variants were not successfully benchmarked on CIFAR-10 and that SMC-GMJE scores 74.52% against MoCo v2 79.02% at 200 epochs (Table 1).

  4. Presentation. We will reduce repeated claims, distinguish parametric GMJE from the non-parametric SMC branch, and qualify the InfoNCE connection and unconditional generation claims. We appreciate the opportunity to make the methodological scope and remaining limitations more precise.

Thank you for your answers. I still find my main concerns not fully addressed. The claims are not decisively supported by experiments. The theoretical justification remains weak. Thus, I keep my score.

Thank you for the follow-up. We understand that your remaining concerns are whether the mathematical formulation gives a sound learning objective, and whether the experiments adequately support the proposed framework.

Mathematical scope. The revised manuscript now distinguishes two different issues: the exact Mahalanobis trace cancellation when covariance is estimated from the same batch (Eq. 5), and the separate embedding–covariance scale collapse possible under unconstrained joint likelihood (Sec. 3; Appendix N.4). We agree that decoupling covariance from the batch does not by itself guarantee non-collapse. Importantly, the conditional term −log⁡p(zt∣zc)-\log p(z_t\mid z_c) still supplies paired-view supervision; GMJE-MDN fits targets through context-dependent mixture components (Eq. 9). The formulation therefore retains a cross-view learning signal, but reliable representation dispersion requires additional constraints. We have removed broader anti-collapse claims and qualified the InfoNCE correspondence as an approximation under explicit assumptions.

Experimental scope. The revised paper separates what each experiment demonstrates: GMJE-MDN recovers multiple conditional branches on synthetic data; SMC versus FIFO improves CIFAR-10 linear-probe accuracy 78.08% to 81.99% at fixed M=256M=256 and 1,000 epochs; and MNIST illustrates latent-density sampling using a separate decoder. The CIFAR result tests the non-parametric memory update, not parametric GMJE on a standard image SSL benchmark. Indeed, SMC-GMJE trails MoCo v2 in the separate 200-epoch comparison (74.52% versus 79.02%). We do not present these as decisive evidence of general SSL superiority.

We nevertheless believe the work makes a useful methodological contribution by connecting joint-density conditioning, multi-modal latent prediction, and the qualified contrastive/mixture interpretation, while identifying concrete failure modes that matter for learning the embeddings themselves. We appreciate that broader controlled experiments are needed, and hope the more precise theory and empirical framing help establish the value of this contribution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

manuscript updated (rebuttal version)

We thank all reviewers for their feedback. We have revised the manuscript to clarify its contributions, strengthen the mathematical analysis, and align the claims with the evidence. Changes are highlighted in red in the revised rebuttal version (v2).

  1. Clarified the formulation and method organization (Secs. 1, 4). We revised the Abstract and contributions around joint context–target density modeling, conditional prediction, and the context marginal. A new table distinguishes dual/primal GJE, parametric GMJE-Proto/MDN/GNG, and non-parametric SMC-GMJE, which use different training objectives and have distinct empirical support.

  2. Strengthened related work and explained the context marginal (Sec. 2). We compare GMJE with VJEPA, Var-JEPA, and VICReg. The decomposition −log⁡p(zc,zt)=−log⁡p(zt∣zc)−log⁡p(zc)-\log p(z_c,z_t)=-\log p(z_t\mid z_c)-\log p(z_c) separates paired-view fitting from context-density modeling. The latter's geometric role is explicit, but its individual empirical benefit remains untested.

  3. Clarified the collapse analysis (Sec. 3; Appendix). The Mahalanobis Trace Trap gives an exact d/2d/2 data-fit term when an invertible covariance is estimated from the same unregularized batch. We distinguish this cancellation from co-adaptive embedding–covariance scale collapse. Prototype parameters, conditional fitting, and dispersion controls address different failure modes; none alone ensures non-collapse. We also corrected the claimed equivalence of dual GP and primal joint-Gaussian objectives, retaining only valid Gram-matrix identities.

  4. Qualified the InfoNCE connection (Sec. 4; Appendix). Proposition 1 now specifies normalization, proxy-key, represented-positive, and low-temperature matching-component assumptions for approximating InfoNCE by a Gaussian KDE conditional. General finite-temperature mixture NLL is not identical to InfoNCE, and block-diagonal covariance implies independence within each component, not of the full mixture.

  5. Clarified SMC and its comparisons (Secs. 4.2, 5.2). The weighted contrastive denominator is a surrogate, not an exact nonuniform KDE likelihood; resampling approximates the weighted bank and can duplicate keys. SMC is a query-adaptive alternative to FIFO, not a demonstrated improvement over all memory policies. At M=256M=256 over 1,000 epochs, SMC achieves 81.99% versus FIFO's 78.08%. In the separate 200-epoch benchmark, it scores 74.52% versus MoCo v2's 79.02%; these results do not establish an across-budget ranking.

  6. Clarified experimental scope (Sec. 5). Synthetic tests show GMJE-MDN captures distinct modes while squared-error JEPA averages them. CIFAR-10 tests the non-parametric bank update, not parametric GMJE's competitiveness in image SSL. MNIST demonstrates qualitative latent sampling with a separately trained decoder, not matched high-fidelity generation. All original figures and numerical results are unchanged.

  7. Identified remaining evaluations and improved organization. We explicitly list missing strong SSL baselines (I-JEPA, VICReg, SwAV), marginal and collapse-control ablations, alternative memory policies, multiple seeds, uncertainty checks, and quantitative sampling comparisons as future work. We also refined the appendix and merged Discussion and Conclusion.

  8. Moved the scale-collapse counterexample into the main text (Sec. 3; Appendix N.4). Following em2q's suggestion, we placed it immediately after the Trace Trap and condensed N.4 to focus on remedies and their limits, making the distinction between the two collapse mechanisms clearer.

We believe these revisions strengthen the manuscript's rigor and clarity. To our knowledge, GJE/GMJE offers a first-of-its-kind explicit joint-density modeling framework for JEPA-style representation learning, unifying conditional prediction, context-density geometry, and multi-modal representation modeling. We hope its methodological contribution outweights the current empirical limitations, and thank the reviewers for their constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.