acceptodds
Under review as a conference paper at ICLR 2027

Predictive Self-Supervised Learning Provably Identifies Stochastic Signals under Nuisance

Abstract

Self-supervised learning (SSL) by predicting in latent space, without generating the input data itself, learns highly abstract, useful representations. Intuitively, this success is often attributed to its ability to discard nuisance information that is irrelevant to prediction. However, this poses a conundrum: both stochastic variation in a prediction-relevant latent signal and true nuisance make observations partly unpredictable; how could they be distinguished? Surprisingly, we prove that common SSL methods can achieve exactly this, by implicitly instantiating a latent- variable model with stochastic dynamics and observation-private nuisance. We trace their ability to recover the stochastic signal to two complementary principles: Predictive mutual information maximization ensures that representations retain the information needed for prediction, while latent distribution matching constrains how this information is encoded, thereby making the retained signal identifiable. We confirm this identifiability result in simulations for Gaussian predictors, which recover the true signal up to an affine transformation even in dynamic, nuisance- laden environments.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Non-generative latent variable model, JEPA, Nonlinear ICA, Causal representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4ezqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that, under some reasonable assumptions, when observations are generated from both semantic signal variables and noisy nuisance variables, a self-supervised learning model that maximizes the predictive mutual information and performs latent distribution matching can recover the signal variables up to an affine transformation, although it seems impossible to completely eliminate the effect of the nuisance. Through a series of experiments, the paper demonstrates that the theory roughly holds within the scope of its assumptions.

Critical strengths

The topic of this paper is interesting and significant: whether self-supervised learning models can extract meaningful variables from complex observations while ignoring irrelevant nuisance in the data. This is, in fact, an important goal for world models.

Critical weaknesses
  1. The theoretical contributions need to be strengthened. The core theory of this paper mainly comes from LDM (Mikulasch & Zenke, 2026), iVAE (Khemakhem et al., 2020), and GIN (Sorrenson et al., 2020), and the paper does not even cite GIN. The authors need to clarify their contributions relative to these works. Moreover, following the idea of GIN, for Gaussian prediction with d_z ≥ d_s, the learned model may force d_s of the d_z dimensions to contain all the signal information and push the nuisance information into the remaining d_z − d_s dimensions. Such a result could strengthen the paper's contribution. Furthermore, the current definition of the generative model is not compatible with world models; at the very least, the actions need to be separated from the conditioning variables. Finally, the optimization objective is hard to achieve in practice, so it would be better to analyze approximate identifiability, as has been done for LeJEPA (Klindt et al., 2026).
  2. The experimental evidence is insufficient. The paper only verifies the preliminary effectiveness of the theory within the scope of its assumptions, without considering cases outside these assumptions. I think the authors should at least test Assumptions 1 and 4. The paper should also compare with other self-supervised models with identifiability guarantees, such as LeJEPA with SIGReg (Balestriero & LeCun, 2025) and DynCL with InfoNCE (Laiz et al., 2025).

References

  • Balestriero & LeCun (2025). LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. arXiv:2511.08544.
  • Khemakhem et al. (2020). Variational Autoencoders and Nonlinear ICA: A Unifying Framework. AISTATS 2020.
  • Klindt, LeCun & Balestriero (2026). When Does LeJEPA Learn a World Model? arXiv:2605.26379.
  • Laiz, Schmidt & Schneider (2025). Self-Supervised Contrastive Learning Performs Non-Linear System Identification. ICLR 2025.
  • Mikulasch & Zenke (2026). Understanding Self-Supervised Learning via Latent Distribution Matching. ICML 2026.
  • Sorrenson, Rother & Köthe (2020). Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN). ICLR 2020.
Questions for authors
  1. The identifiability results of iVAE and GIN are both component-wise, while yours is only affine. Can your results be improved by adding independence assumptions on the variables? In addition, Assumption 4 seems quite weak; could you explain its specific meaning?
  2. In the theory of LeJEPA (Klindt et al., 2026), the transformation matrix appears to have additional properties beyond invertibility. Can a similar property be derived from your theory?
8 comments

We thank the reviewer for their positive assessment of the importance of the work.

Regarding weaknesses:

  1. Theoretical Contribution

We thank the reviewer for pointing out the results of GIN, which are indeed relevant for presented results and we will discuss them in the introduction and discussion sections. GIN enables to separate between condition-dependent signal and condition-independent nuisance variables. The presented work mainly differs from GIN in that our model does not rely on an invertible encoder, and does not have to retain the nuisance in the latent variables. On the other hand, GIN recovers the signal component-wise in a permuted subset of latent dimensions, if the covariance of the Gaussian predictor changes based on the context. The same could indeed be achieved also in our model, with the same mechanism. We will discuss this in the main text and provide a corollary extending this result to our model.

“The current definition of the generative model is not compatible with world models” For simplicity, we define the context as containing all conditioning variables, including actions, as shown in Figure 1. We will make this more clear also in the main text.

“It would be better to analyze approximate identifiability” This is indeed a worthwhile, although non-trivial extension of the theory. We find that this requires additional assumptions. LeJEPA achieves this with the strong assumption of Gaussian latent marginals and a simple alignment loss. We have added a theorem to the appendix that derives an analogous approximate recovery theorem under additional assumptions. We nevertheless consider the empirical experiments the main validation of the theory, since the additional assumptions in this new theorem remain untested for our applications. Further, these are the weakest assumptions that we found sufficient to prove the desired result, but it is possible that approximate recovery is still achievable even if these assumptions are violated.

  1. Experimental evidence

Additionally to the Hopper experiment that violates the Gaussian prediction assumption, we now also test how generalized-Gaussian relations affect learning, as detailed in the appendix. Specifically, we test both long- and short-tailed conditional distributions in the setup of our first experiment. The results show remarkable robustness of the methods to these deviations from the assumptions.

Regarding the suggestion to violate Assumptions 1 and 4: We would like to clarify that Assumption 1 defines nuisance as conditionally independent of the predictive context given the signal. Violating it introduces predictive information into nuisance variables, which could then be considered part of the signal. Similarly, violating Assumption 4 makes some signal direction unpredictable, effectively turning it into nuisance. Strict violations of these assumptions therefore correspond to settings already explored in our experiments.

Nevertheless, we additionally tested soft deviations from both assumptions. Introducing weakly predictable nuisance (Assumption 1) showed robust recovery that degraded with increasing nuisance predictability. Reducing signal predictability (approaching a violation of Assumption 4) led to gradually declining recovery for most methods, while logdet exhibited an unexpected early collapse. This behavior is consistent with our new approximate recovery theorem, which predicts weaker recovery guarantees as condition diversity decreases. We report these additional results in the Appendix.

We have also conducted preliminary comparisons with the objectives of DynCL and LeJEPA, using the same encoder and predictor architecture as in our experiments. Neither approach achieved reliable signal recovery in the tested configurations. However, these comparisons require important qualifications.

  • DynCL was originally developed with linear or piecewise linear predictors. Our adaptation with a more flexible nonlinear predictor resulted in representation collapse.
  • LeJEPA uses direct latent alignment with Gaussian marginal regularization, rather than the predictive likelihood considered here. The theory by Klindt et al also does not accommodate nuisance. Our experiments with a nonlinear predictor and strong nuisance did not yield reliable recovery.

Because these experiments modify the original methods and evaluate them outside their theoretical settings, we refrain from interpreting their failure as evidence of inferior performance as methods for nonlinear systems identification in general.

Questions:

  1. See answer to weakness 1 above. For the second question see answer 1 to reviewer ByCo.

  2. We assume the reviewer is referring to the transformation matrix being orthogonal. This requires additional assumptions. LeJEPA achieves this by assuming identity covariance (Σ=I\Sigma=I) Gaussian marginals for both true and recovered variables. Under this strong assumption this result would also follow for our model.

Thank you for the response and the revised manuscript. Could you please upload a version in which all changes are highlighted in color?

We have added a pdf with changes highlighted to the supplementary material.

Reviewer_ByCoOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  • Predictive self-supervised learning methods such as InfoNCE, SimCLR, and VICReg are assumed to distinguish stochastically varying signal from completely unpredictable nuisance, and model only the former signal. However, no prior work has formalized this distinction within a statistical framework.This paper theoretically shows why this distinction is possible.
  • It writes the objective function common to these methods as InfoLDM (predictive mutual-information maximization + latent distribution matching). The key point is the simultaneous realization of MI maximization and LDM. Under ideal conditions, MI maximization requires "no information necessary for predicting the signal is dropped", and LDM requires "the shape of p_theta is matched to the true predictive conditional, fixing the way information is encoded down to an affine-transformation ambiguity". With MI alone, z retains ambiguity under arbitrary invertible transformation. LDM constrains p_theta to an exponential family and imposes distribution matching. Then, identification up to an affine transformation becomes possible. Further, if we assume d_z = r_s, there remains no capacity for nuisances, so they are discarded from the learned model.
  • Experiments use three types of simulation data, verifying the validity and limitations of the theory. It also confirms that the type of entropy estimator affects the practical performance.
Critical strengths
  1. The formulation is concise and relatively easy to understand.
  • The authors explicitly describe a nuisance orthogonal to the signal as a graphical model, and give formal backing for why stochastic variation arising from signal dynamics can be separated from unpredictable noise. That has so far remained "intuition" in the literature.
  1. The authors honestly demonstrate the limits of the theory; e.g., degraded recovery accuracy for high-dimensional signals.
Critical weaknesses
  1. Assumptions 3 and 4 are too strong.
  • The true signal dimension r_s cannot be known in practice, yet the theory requires knowing it. Under a bottleneck case d_z < r_s, MI saturation is impossible, which takes the theorem out of scope. On the other hand, with d_z > r_s the signal is guaranteed to be recovered, but there is no "force" to exclude nuisance from leaking in.
  • Further, Assumption 3 requires KL=0 (successful global optimization + the used model covers the true distribution). This is unrealistic under finite-sample, local optimization training.
  • In other words, the theorem shows "what is recovered if success is achieved, assuming success", and is slightly distant from a theoretical foundation for practical use.
  1. There is no experimental validation on real data.
  • Readers are not sure (so it is interesting) how accurately nuisance can be separated in generic situations where Assumption 3 does not hold. The provided synthetic experiments set d_z = r_s as known, following the theoretical regime.
  • The behavior under realistic settings outside the assumption is unknown from the current manuscript, leaving some doubts on applicability to real model evaluations.
Questions for authors
  • Is it possible to (approximately) evaluate robustness in situations where the assumptions are violated? For example, d_z is not equal to r_s, non-Gaussian predictive distributions, or entropy estimation errors at finite samples.
  • Concerning Table 1, R^2 decreases as d_s (= d_z) is increased to 15 and 20; is the cause optimization failure, entropy-estimation error, or something overlooked by the theory? I would appreciate an analysis if available.
  • Related to the two questions above, can you propose criteria for which entropy estimator (which SSL method) to use depending on the task/data? I believe such guidance would be welcomed by practitioners.
3 comments

We thank the reviewer for the positive assessment of the importance of the work and of the manuscript presentation, informativeness, and soundness.

Regarding the weaknesses:

  1. “Assumptions 3 and 4 are too strong.”

We agree that Assumption 3 is strong, which we discuss below. Assumption 4, however, is a standard condition-diversity requirement in identifiability theory (e.g., Khemakhem et al., 2020). It ensures that the conditioning variables provide sufficient variation to distinguish all dimensions of the signal. Without this assumption full identification via a predictive model is impossible in general. In the Gaussian setting we considered, it simply requires the conditional means to vary in all signal directions. We therefore consider it a relatively mild identifiability condition.

"The true signal dimension r_s cannot be known in practice, yet the theory requires knowing it.” This appears to be a misunderstanding. Our theory does not require prior knowledge of the signal dimension. The main theorem explicitly allows overparameterized representations with rz≥rsr_z\geq r_s. We also demonstrate this experimentally in the Causal3DIdent and Hopper experiments, both of which use representations with more dimensions than the underlying signal. Our results show that in practice models should be overparametrized if the true dimension is unknown, to avoid the problem of failed recovery. We will clarify this point in the manuscript.

"Assumption 3 requires KL=0” This is indeed a strong assumption, but one that is the common starting point of most identifiability results. We demonstrate the relevance for the finite sample case in our experiments. As requested also by reviewer 4ezq, we have further derived an approximate recovery result which we now include in the appendix.

  1. Lack of evaluation on real-world data

We agree that evaluating our theory on real-world data would be valuable. However, a meaningful validation of our identifiability guarantees requires access to ground-truth signal and nuisance variables as defined in our theoretical framework. To our knowledge, no suitable real-world dataset is available, although we would welcome suggestions. This difficulty is also reflected in previous identifiability studies with a similar scope (e.g., Zimmermann et al., 2021; von Kügelgen et al., 2021), which rely on controlled experiments to validate their theoretical predictions. Importantly, our contribution is theoretical rather than methodological: the SSL methods we analyze have already been evaluated on real-world data in previous work, e.g., SimCLR (Chen et al., 2020), Mikulasch et al. 2026 or others. Applying these established methods to additional real-world datasets without access to the underlying latent variables would therefore provide little new evidence regarding our theoretical claims. We instead focus on controlled experiments that allow us to directly assess identifiability under nuisance.

Questions:

  1. We explicitly present results for rz≠rsr_z\neq r_s as mentioned above. We further tested non-Gaussian conditional distributions (long and short tail) in the setup of our first experiment, which demonstrate robustness of the methods. We report in these additional results in the appendix.

  2. Degraded performance likely comes from two sources: Curse of dimensionality (exponentially many samples required for similar coverage), and increase in entropy estimation difficulty. However, after fixing the implementation error pointed out by reviewer Cs5j, performance does not degrade for all estimators in the same way.

  3. This is a good suggestion. In general, entropy estimators come with a bias-variance trade-off. E.g., logdet assumes Gaussian marginals, but is more sample efficient than kNN and KDE. However, in the revised manuscript we also introduce an additional estimation method based on Stein entropy gradient estimation, which does not make strong assumptions on the disribution and consistently outperforms other methods, likely because it omits explicitly constructing the latent marginal distribution. We will discuss this in the Conclusion section.

To authors; thank you for the thoughtful rebuttal and revision. Your input effectively resolve most of my concerns.

Especially,

  • In response to the criticism of Assumption 3 (KL = 0), the authors have added Theorem 2, which guarantees approximate recovery.
  • Regarding the knowledge of r_s, I acknowledge the authors' clarification: the theorem does not require prior knowledge of r_s (it is instead derived as a conclusion of the theorem), and my original phrasing was inaccurate in this regard. Thank you for correcting my misunderstanding.
  • Regarding Q3, the guidance that Stein — which does not assume Gaussianity of the latents — is the preferred general-purpose choice is consistent with the experimental results (Tables 1, 5).
  • Lack of real-data validation: I accept this limitation is shared by identifiability studies of this kind.

Still I have two points to note. Both are limited in scope, and I would appreciate any response.

  1. On the margin of overparameterization. I have confirmed that the theory permits d_z ≥ r_s and that the current experiments are indeed overparameterized. However, the margins are all close to the true signal dimensions in the experiments. I think what is demonstrated here is operation under 'minimal' overparameterization. In practice, r_s is unknown, so users would choose a much larger d_z for safety; such kind of realistic regime is untested in more difficult datasets. Also, a large d_z may itself aggravate entropy estimation in a high-dimensional, sparse latent space (a resurgence of the estimator problem shown in Table 1). This raises a concern that the practical operating region is squeezed from both sides: underparameterization (outside the theorem's scope) on one side, and estimator failure on the other.
    Relatedly, could you clarify in the manuscript that overparameterization is recommended when r_s is unknown? The rebuttal mentions this clarification, but I could not find it in the revised manuscript: excuse me if I simply overlooked.

  2. On the realism of Assumption 6. Assumption 6 (the condition γ > 0 on the curvature of the interpolated context posteriors), introduced for Theorem 2, is intuitively understandable as requiring that score discrepancies do not concentrate on a small set of contexts and become undetectable. However, I have no sense of how well it holds in practice.


Overall, this revision is sincere and substantive, and the response to the main criticism (KL = 0) constitutes theoretical progress.

We thank the reviewer for their positive and helpful reply. We are happy to answer the remaining points.

  1. On the margin of overparameterization. This is useful and insightful to test. We provide two additional experiments in the appendix, where we increase the latent dimensionality of the model trained on the 5D synthetic experiment, and the Hopper. We find that on the synthetic experiment recovery stays high even for highly overcomplete 128D latents. For the Hopper we find that the readout actually becomes better the higher-dimensional the representation is; likely because the Hopper variables are slightly non-linearly encoded, as we discuss there. This shows that also in for high-dimensional model latents estimators continue to perform well, while estimator failures are mostly attributable to high-dimensional signal variables. We have added an explicit recommendation together with the discussion of the Hopper results in the main text.

  2. We agree with reviewers ByCo and 4ezq that it is not straightforward to assess how well Assumption 6 is satisfied in practice; it is challenging to test and not intuitive. To mitigate this point, we decided to replace the newly added general approximate recovery result with a Theorem for Gaussian dynamics for which the assumptions are more tangible. While this result is less general we think it is actually more useful and interpretable, since it is more closely connected to the experimental results with Gaussian dynamics, and it allows us to sidestep assumptions whose practical relevance is hard to judge. As a consequence, the average recovery error is now bounded by a combination of the goal-function gap to the optimum, and the predictability of the signal (given by the prediction covariance).

We have updated the manuscript accordingly and hope these changes address the remaining concerns of the reviewer.

Reviewer_Cs5jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies when predictive self-supervised learning can recover a stochastically evolving signal from observations containing private nuisance variables. Its population objective combines predictive mutual information with latent distribution matching. Under information saturation, exact distribution matching, exponential-family conditionals, and condition diversity, Theorem 1 establishes an affine readout of the true sufficient statistic from the learned statistic. Fixed-covariance Gaussian conditionals yield affine recovery of the signal itself. This paper provides synthetic sequences, a stochastic Causal3DIdent construction, and a noisy Hopper environment to illustrate recovery and nuisance retention with different entropy estimators.

Critical strengths
  • Theorem 1 accommodates stochastic signal transitions and observation-private nuisance simultaneously. The proof uses mutual-information saturation to establish the conditional independences needed for likelihood-ratio matching, then derives a full-rank affine statistic readout. This provides a precise answer to the paper’s motivating question.
Critical weaknesses
  1. Predictor bottleneck. Appendix A.3 and the code use a 10-dimensional recurrent state with a linear prediction head, including for 15- and 20-dimensional signals. The predicted means therefore have rank at most 10. This confounds attributing the recovery decline in Table 1 to entropy estimation or optimization.
  2. Entropy comparisons change loss weighting. The supplied kNN term omits the dimension multiplier and clips neighbour distances at the 90th percentile, while prediction error is summed over coordinates. Differences from log-determinant entropy may therefore reflect weighting and clipping as well as estimation quality. This needs clarification but does not establish a failure of the recovery theorem.
Questions for authors
  1. See weaknesses.
  2. Was the 10-dimensional LSTM and linear prediction head used for every Table 1 setting? If so, do you agree that the 15- and 20-dimensional settings violate realizability, and how should their recovery decline be interpreted? Existing results with a sufficiently wide predictor would be especially informative if available.
  3. Which of the implemented entropy estimators preserve the population InfoLDM objective exactly, and which only approximate it under additional distributional assumptions? Distinguish the theorem’s guarantee from the empirical behavior of those surrogates.
1 comment

We thank the reviewer for the positive assessment and the in-depth technical feedback.

Regarding the weaknesses:

  1. We thank the reviewer for spotting this error, which was an oversight in our implementation. We now are consistently using 2N-dimensional recurrent states, where N is the dimension of the latent space. Correcting this improved the performance in higher dimensions, particularly for the logdet estimator.

  2. The dimensionality factor is indeed missing for correctly scaled entropy estimation. We corrected the implementation, but this did not change any of the presented results, since it amounts to a simple rescaling of the representation. It is important if one aims to sample from the latent transition model, since then the representation and the transition noise have to be correctly scaled with respect to each other. Clipping empirically improves learning stability and has been proposed previously by Mikulasch et al, 2026.

Questions:

For 1 and 2 see above.

  1. The correctly scaled kNN estimator is exact in the limit, kDE is also exact in the limit if the bandwidth is scaled accordingly. The logdet estimator assumes Gaussian distributed latent marginals. We will add this information more clearly to the text.

Comments on the paper

1 comment
AuthorsOpenReview ↗

First revision

We thank the reviewers for their suggestions, and for commending the significance, soundness, and clarity of the manuscript. The main concerns the reviewers noted were strong assumptions of the theory and insufficient experimental evaluation. We addressed them in the following ways:

Strong assumptions: Reviewers pointed out the strength of Assumption 3, successful global optimization. We now provide an approximate recovery theorem under incomplete matching and mutual information maximization, additionally to the numerical evaluation of our theory. This establishes also in theory that perfect optimization is not strictly required for approximate recovery of latent variables.

Insufficient experimental validation: We now test several violations of our theoretical assumptions in the data. Specifically, we test recovery under non-Gaussian noise in the dynamics, and demonstrate robustness to this violation. We also tested soft violations of the assumptions where possible, such as making nuisance signals weakly predictable, or reducing the predictability of the signal variables.

We hope these additions answer the main questions and concerns of the reviewers. We provide an updated manuscript which contains the new results and updated text. For the final version we will integrate the new results more tightly into the narrative of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.