- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper shows that, under some reasonable assumptions, when observations are generated from both semantic signal variables and noisy nuisance variables, a self-supervised learning model that maximizes the predictive mutual information and performs latent distribution matching can recover the signal variables up to an affine transformation, although it seems impossible to completely eliminate the effect of the nuisance. Through a series of experiments, the paper demonstrates that the theory roughly holds within the scope of its assumptions.
Critical strengths
The topic of this paper is interesting and significant: whether self-supervised learning models can extract meaningful variables from complex observations while ignoring irrelevant nuisance in the data. This is, in fact, an important goal for world models.
Critical weaknesses
- The theoretical contributions need to be strengthened. The core theory of this paper mainly comes from LDM (Mikulasch & Zenke, 2026), iVAE (Khemakhem et al., 2020), and GIN (Sorrenson et al., 2020), and the paper does not even cite GIN. The authors need to clarify their contributions relative to these works. Moreover, following the idea of GIN, for Gaussian prediction with d_z ≥ d_s, the learned model may force d_s of the d_z dimensions to contain all the signal information and push the nuisance information into the remaining d_z − d_s dimensions. Such a result could strengthen the paper's contribution. Furthermore, the current definition of the generative model is not compatible with world models; at the very least, the actions need to be separated from the conditioning variables. Finally, the optimization objective is hard to achieve in practice, so it would be better to analyze approximate identifiability, as has been done for LeJEPA (Klindt et al., 2026).
- The experimental evidence is insufficient. The paper only verifies the preliminary effectiveness of the theory within the scope of its assumptions, without considering cases outside these assumptions. I think the authors should at least test Assumptions 1 and 4. The paper should also compare with other self-supervised models with identifiability guarantees, such as LeJEPA with SIGReg (Balestriero & LeCun, 2025) and DynCL with InfoNCE (Laiz et al., 2025).
References
- Balestriero & LeCun (2025). LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. arXiv:2511.08544.
- Khemakhem et al. (2020). Variational Autoencoders and Nonlinear ICA: A Unifying Framework. AISTATS 2020.
- Klindt, LeCun & Balestriero (2026). When Does LeJEPA Learn a World Model? arXiv:2605.26379.
- Laiz, Schmidt & Schneider (2025). Self-Supervised Contrastive Learning Performs Non-Linear System Identification. ICLR 2025.
- Mikulasch & Zenke (2026). Understanding Self-Supervised Learning via Latent Distribution Matching. ICML 2026.
- Sorrenson, Rother & Köthe (2020). Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN). ICLR 2020.
Questions for authors
- The identifiability results of iVAE and GIN are both component-wise, while yours is only affine. Can your results be improved by adding independence assumptions on the variables? In addition, Assumption 4 seems quite weak; could you explain its specific meaning?
- In the theory of LeJEPA (Klindt et al., 2026), the transformation matrix appears to have additional properties beyond invertibility. Can a similar property be derived from your theory?
8 comments
Thank you for the response and the revised manuscript. Could you please upload a version in which all changes are highlighted in color?
We have added a pdf with changes highlighted to the supplementary material.
We thank the reviewer for their positive assessment of the importance of the work.
Regarding weaknesses:
We thank the reviewer for pointing out the results of GIN, which are indeed relevant for presented results and we will discuss them in the introduction and discussion sections. GIN enables to separate between condition-dependent signal and condition-independent nuisance variables. The presented work mainly differs from GIN in that our model does not rely on an invertible encoder, and does not have to retain the nuisance in the latent variables. On the other hand, GIN recovers the signal component-wise in a permuted subset of latent dimensions, if the covariance of the Gaussian predictor changes based on the context. The same could indeed be achieved also in our model, with the same mechanism. We will discuss this in the main text and provide a corollary extending this result to our model.
“The current definition of the generative model is not compatible with world models” For simplicity, we define the context as containing all conditioning variables, including actions, as shown in Figure 1. We will make this more clear also in the main text.
“It would be better to analyze approximate identifiability” This is indeed a worthwhile, although non-trivial extension of the theory. We find that this requires additional assumptions. LeJEPA achieves this with the strong assumption of Gaussian latent marginals and a simple alignment loss. We have added a theorem to the appendix that derives an analogous approximate recovery theorem under additional assumptions. We nevertheless consider the empirical experiments the main validation of the theory, since the additional assumptions in this new theorem remain untested for our applications. Further, these are the weakest assumptions that we found sufficient to prove the desired result, but it is possible that approximate recovery is still achievable even if these assumptions are violated.
Additionally to the Hopper experiment that violates the Gaussian prediction assumption, we now also test how generalized-Gaussian relations affect learning, as detailed in the appendix. Specifically, we test both long- and short-tailed conditional distributions in the setup of our first experiment. The results show remarkable robustness of the methods to these deviations from the assumptions.
Regarding the suggestion to violate Assumptions 1 and 4: We would like to clarify that Assumption 1 defines nuisance as conditionally independent of the predictive context given the signal. Violating it introduces predictive information into nuisance variables, which could then be considered part of the signal. Similarly, violating Assumption 4 makes some signal direction unpredictable, effectively turning it into nuisance. Strict violations of these assumptions therefore correspond to settings already explored in our experiments.
Nevertheless, we additionally tested soft deviations from both assumptions. Introducing weakly predictable nuisance (Assumption 1) showed robust recovery that degraded with increasing nuisance predictability. Reducing signal predictability (approaching a violation of Assumption 4) led to gradually declining recovery for most methods, while logdet exhibited an unexpected early collapse. This behavior is consistent with our new approximate recovery theorem, which predicts weaker recovery guarantees as condition diversity decreases. We report these additional results in the Appendix.
We have also conducted preliminary comparisons with the objectives of DynCL and LeJEPA, using the same encoder and predictor architecture as in our experiments. Neither approach achieved reliable signal recovery in the tested configurations. However, these comparisons require important qualifications.
Because these experiments modify the original methods and evaluate them outside their theoretical settings, we refrain from interpreting their failure as evidence of inferior performance as methods for nonlinear systems identification in general.
Questions:
See answer to weakness 1 above. For the second question see answer 1 to reviewer ByCo.
We assume the reviewer is referring to the transformation matrix being orthogonal. This requires additional assumptions. LeJEPA achieves this by assuming identity covariance (Σ=I) Gaussian marginals for both true and recovered variables. Under this strong assumption this result would also follow for our model.