acceptodds
Under review as a conference paper at ICLR 2027

SplitJEPA: Learning Invariant and Variant Latent Worlds without Reconstruction

14 views

Abstract

Understanding a dynamical world calls for more than a latent state that summarizes its observations: the state should also be organized into the factors that stay shared across related observations and the factors that vary between them. For example, a robot pushing a cube to a goal should take the same action when the camera shifts or the lights dim, since nothing in the scene has moved. Existing approaches to this decomposition commonly obtain it through reconstruction, so the latent variables must first explain the entire observational world before their organization can be trusted. Joint embedding predictive architectures (JEPAs) model the latent state directly and never reconstruct, yet no existing result recovers the invariant and variant parts of the state they learn. How to learn the invariant-variant structure of the latent world without paying for its reconstruction therefore remains open. To close this gap, we introduce SplitJEPA, a JEPA that jointly recovers the latent state and its invariant and variant organization directly in representation space, without any reconstruction. We prove that, under stationary Gaussian predictive dynamics and a full-rank variation condition, \modelname identifies the invariant and variant subspaces up to independent block-wise isometries, without introducing an observation decoder. Since the guarantee needs no decoder, the result extends reconstruction-free latent recovery to invariant-variant block identification. Experiments on synthetic nonlinear systems and robotic manipulation tasks support the theoretical results and show their practical value for both robustness and efficiency.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aDimOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SplitJEPA, a reconstruction-free self-supervised learning framework that aims to separate invariant and variant latent representations. The method combines predictive learning to identify the full latent state with invariant correspondences to separate the two latent blocks. The authors provide theoretical identifiability guarantees under specific assumptions, including Gaussian predictive dynamics, exact whitening, and sufficient variation in invariant pairs. The approach is evaluated through synthetic experiments examining block recovery and scalability, and robotic manipulation tasks (PickCube and PushCube) assessing downstream performance and out-of-distribution generalisation.

Critical strengths
  • The paper addresses an interesting and important problem of learning separate invariant and variant representations without reconstructing observations. This is relevant to representation learning and OOD generalisation.

  • The proposed framework combines predictive learning and invariant correspondences to address latent identifiability. The theoretical analysis provides an interesting perspective on how different sources of information can help resolve latent ambiguity.

  • The paper provides both theoretical analysis and experimental validation using synthetic data and robotic manipulation tasks. The experiments show that the invariant representation is less sensitive to visual variations and demonstrate promising performance under visual distribution shifts, even with limited training data.

  • The paper reports improved computational and sample efficiency compared with a reconstruction-based baseline. The ablation studies also help demonstrate the contributions of different components to block recovery and representation learning.

Critical weaknesses
  1. Strong assumptions behind the theoretical guarantees The theoretical results rely on specific assumptions, including Gaussian predictive dynamics, exact whitening, and sufficiently diverse invariant correspondences. While these assumptions support the theoretical analysis, it remains unclear how realistic they are for practical applications. In particular, real-world latent dynamics may not follow the assumed Gaussian structure, and the underlying latent factors are generally unknown. It is therefore difficult to determine to what extent the theoretical identifiability guarantees apply beyond the considered settings.

  2. Limited validation of invariant and variant block separation The paper demonstrates block recovery in synthetic experiments and evaluates the sensitivity of the learned representations in robotic tasks. However, it remains unclear whether the invariant and variant representations are correctly separated in the robotic experiments. Showing that the invariant block is less sensitive to certain variations does not necessarily establish that the two blocks capture distinct latent factors. Additional analysis would be helpful to establish whether the practical representations exhibit the block separation claimed by the theoretical framework.

  3. Limited task complexity and scalability The robotic evaluation is limited to two relatively simple manipulation tasks, PickCube and PushCube. While these tasks provide useful initial validation, it is unclear whether the proposed method can scale to more complex tasks involving multiple objects, more complicated interactions, or more diverse dynamics. The synthetic experiments also consider relatively low-dimensional latent spaces. Further evaluation would help establish whether the method remains effective in more challenging and higher-dimensional settings.

  4. Limited evaluation of OOD generalisation The OOD experiments mainly consider camera and lighting changes. However, OOD conditions can arise from different sources, including changes in object properties, geometry, dynamics, or task configurations. It remains unclear whether the proposed invariant representation would remain robust under these different types of shifts. In addition, the results in Table 2 show that the invariant representation does not consistently outperform the full representation, raising questions about when the learned invariant features are beneficial for downstream tasks.

  5. Practical construction of invariant correspondences The theoretical framework assumes that invariant pairs share the same invariant factors while the variant factors change sufficiently. However, the robotic experiments construct correspondences using approximately aligned trajectories. It is unclear whether such correspondences actually preserve the same invariant latent factors. Since the theoretical guarantees depend on this assumption, additional clarification or analysis is needed to understand how correspondence errors affect the learned representations.

  6. Normalisation and its impact on block identifiability The theoretical guarantees assume exact whitening of the learned representations, while practical training relies on variance regularisation and ZCA whitening. Although the paper investigates the effect of sample size on whitening and block recovery, it remains unclear how sensitive the method is to the training batch size and imperfect covariance estimation. Since whitening is central to the theoretical analysis, further investigation would help establish the robustness of the method under practical training conditions.

Questions for authors
  1. Can the authors clarify how the Gaussian predictive dynamics assumption can be justified for real-world datasets, particularly when the underlying latent factors and dynamics are unknown?

  2. How do the authors verify that the learned invariant and variant representations are correctly separated in the robotic experiments, beyond measuring their relative sensitivity to viewpoint or temporal changes?

  3. Could the authors clarify how predictive and invariant pairs can be reliably selected without access to the underlying latent factors? In particular, how sensitive is block recovery to errors in invariant correspondences, and how can the sufficient variation condition be assessed when matched pairs are constructed using approximately aligned trajectories?

  4. Given that the theoretical guarantees rely on exact whitening and global optimisation, could the authors clarify to what extent these conditions are satisfied in practice? In particular, how is covariance estimation performed, and does the training batch size affect subsequent block identifiability?

  5. The results in Table 2 show that using only the invariant representation does not consistently outperform using the full representation. Could the authors clarify when invariant representations are expected to improve downstream performance and when variant representations may still contain useful task-related information?

  6. Can the authors discuss how the method would scale to more complex tasks and higher-dimensional latent spaces? In particular, would the block recovery mechanism remain effective when there are more interacting latent factors?

  7. How well would the proposed method generalise to other types of OOD shifts beyond camera and lighting changes, such as changes in object properties, geometry, or dynamics?

7 comments
AuthorsOpenReview ↗

We sincerely appreciate your time and detailed feedback (1/4)

Dear Reviewer aDim,

We sincerely appreciate your time and detailed feedback. Your suggestions are helpful for clarifying the scope of the guarantees and strengthening the practical analysis. Please find our point-by-point responses below.


Q1. How can Gaussian predictive dynamics be justified for practical datasets when the underlying latent factors and dynamics are unknown? A1. Thank you for raising this important point. Our theoretical guarantee is conditional on the model stated in Section 2, Eq. (3) and the normalization conditions in Section 3.4, Eq. (11).

As explained in L195–215 and L277–286, the argument separates recovery of the complete latent state from identification of its partition. The Gaussian predictive model establishes the first step through the global recovery result of Klindt et al. [1]. The second step uses the recovered geometry and the matched correspondences. An invariant pair satisfies ΔzH=0\Delta z_H=0, and therefore ΔrH=A12ΔzL\Delta r_H=A_{12}\Delta z_L. If ΔrH=0\Delta r_H=0 and the variant differences have a full-rank second moment, then A12=0A_{12}=0. Orthogonality subsequently gives A21=0A_{21}=0. This is the argument summarized in L256–261 and proved in Appendix C.2, with the final orthogonality step in L812–823.

The partition argument depends on global orthogonal recovery, shared-factor preservation, and sufficient variation. Accordingly, it can be reused for another predictive model if the same global recovery property can be established. In our work, the Gaussian model supplies that preceding result.

We also conduct sensitivity experiments beyond the exact assumptions. As detailed in our response to Reviewer fSrW, Q2, recovery remains stable across the tested non-Gaussian stationary AR distributions and moderate coordinate-dependent correlations; greater heterogeneity degrades separation. These experiments provide empirical evidence within the tested settings. Establishing comparable predictive recovery guarantees for broader practical dynamics remains future work to explore.

[1] When Does LeJEPA Learn a World Model? arXiv, 2026

We have clarified this scope in Section 3.2, L284–286: the correspondence argument applies to another predictive model if the same global recovery property can be established. We also report the non-Gaussian and coordinate-dependent-correlation experiments in Appendix D.5 and Figure 13, which are empirical sensitivity results.


Q2. How is robotic block separation assessed beyond relative sensitivity to viewpoint or temporal changes?

A2. Thank you for this suggestion. We agree that lower sensitivity alone does not determine which latent factors a representation retains. The original analysis in Figure 16 and Appendix E.2 (L1347–1349 and L1372–1381) measures the blocks’ relative responses to matched changes. To complement it, we provide simulator-ground-truth linear probes on both tasks.

The encoder is frozen, and each state is rendered under three camera azimuths and three lighting levels. We fit ridge regressions using disjoint matched-task groups for training, validation, and testing, selecting regularization on the validation groups. The results are reproduced below for convenience; additional discussion is provided in our response to Reviewer fSrW, Q3.

TaskMetricCubeGoalCameraLighting
PickCuberHr_H0.9160±0.01070.9160 \pm 0.01070.1423±0.00860.1423 \pm 0.00860.1129±0.01370.1129 \pm 0.01370.0379±0.00440.0379 \pm 0.0044
PickCuberLr_L0.8350±0.08200.8350 \pm 0.08200.1501±0.02360.1501 \pm 0.02360.5254±0.00700.5254 \pm 0.00700.0624±0.00770.0624 \pm 0.0077
PickCube[rH,rL][r_H,r_L]0.9317±0.01440.9317 \pm 0.01440.1699±0.01380.1699 \pm 0.01380.5716±0.04250.5716 \pm 0.04250.0818±0.01400.0818 \pm 0.0140
PushCuberHr_H0.6582±0.04190.6582 \pm 0.04190.6105±0.05480.6105 \pm 0.05480.0145±0.06710.0145 \pm 0.06710.0572±0.03040.0572 \pm 0.0304
PushCuberLr_L0.6575±0.02250.6575 \pm 0.02250.6029±0.02250.6029 \pm 0.02250.0268±0.02010.0268 \pm 0.02010.0993±0.00890.0993 \pm 0.0089
PushCube[rH,rL][r_H,r_L]0.7797±0.01780.7797 \pm 0.01780.7486±0.02810.7486 \pm 0.02810.0705±0.10770.0705 \pm 0.10770.1788±0.01800.1788 \pm 0.0180

On PickCube, rHr_H retains strong cube-position information, while camera azimuth is more accurately decoded from rLr_L. Under fixed physical state and lighting, signed viewpoint differences also yield higher decoding accuracy from rLr_L. These results support task-state/viewpoint specialization. On PushCube, both blocks retain task-state information, and their concatenation improves decoding. We therefore interpret these results as complementary task-state information across the representation. Together, the probes provide evidence about information retention and factor selectivity beyond the sensitivity analysis in Figure 16.

We have added these ground-truth evaluations to the updated manuscript in Appendix E.2, L1382–1388, and Table 10, alongside the relative-sensitivity analysis in Figure 16.

AuthorsOpenReview ↗

We sincerely appreciate your time and detailed feedback (2/4)

Q3. How are pairs selected without access to the latent factors, how sensitive is recovery to correspondence errors, and how can sufficient variation be assessed?

A3. Thank you for these helpful questions. As defined in Section 2, Eq. (4), invariance is relative to the pairing relation. The practical construction is described in Appendix E.1. We clarify the two types of correspondence and their limitations below.

  • Predictive pairs are adjacent observations within the same trajectory.
  • Cross-view invariant pairs are synchronized renders of the same physical state from different viewpoints. They preserve object, goal, and robot states while varying viewpoint. In our training-pair audit, cube and goal positions agree across all cross-view pairs.
  • Cross-trajectory invariant pairs share known task configurations and are approximately aligned by execution progress. Goal positions agree across these pairs, but instantaneous physical states, including current cube position, can differ. If those instantaneous quantities belong to the intended invariant factors, their differences constitute correspondence errors.

Thus, the two constructions provide different correspondence strengths, as already noted in L1177–1186. Cross-view pairs are state-matched; cross-trajectory pairs broaden execution variation while relaxing exact state alignment.

For sensitivity to correspondence errors, we evaluate controlled violations of the invariant relation while preserving the variant intervention. The additional synthetic results are:

Pairing conditionRH2R_H^2RL2R_L^2Purity∥A12∥F\lVert A_{12}\rVert_F∥A21∥F\lVert A_{21}\rVert_F
Correct0.9979±0.00170.9979 \pm 0.00170.9954±0.00460.9954 \pm 0.00460.9992±0.00100.9992 \pm 0.00100.0344±0.02300.0344 \pm 0.02300.0543±0.04520.0543 \pm 0.0452
5% errors0.9976±0.00160.9976 \pm 0.00160.9951±0.00390.9951 \pm 0.00390.9990±0.00080.9990 \pm 0.00080.0420±0.02050.0420 \pm 0.02050.0678±0.03620.0678 \pm 0.0362
20% errors0.9953±0.00220.9953 \pm 0.00220.9909±0.00270.9909 \pm 0.00270.9970±0.00110.9970 \pm 0.00110.0688±0.01430.0688 \pm 0.01430.1386±0.02220.1386 \pm 0.0222
Correct ; only 2/4 variant directions vary0.9817±0.01540.9817 \pm 0.01540.9838±0.01530.9838 \pm 0.01530.9853±0.01360.9853 \pm 0.01360.2166±0.14480.2166 \pm 0.14480.1962±0.14170.1962 \pm 0.1417

With 5% errors, recovery remains close to the clean-pair result. At 20% errors, purity remains 0.99703, while both leakage terms increase. Moderate mismatch is therefore tolerated in this setting, but increasing errors progressively degrades separation. The final row shows that limited coverage also reduces recovery even with correct pairs. For sufficient variation, Assumption 1 (L248–255) requires changes in every linear direction of the variant space. With approximately aligned trajectories, a practical assessment is to compute paired differences in available execution-state variables, such as joint positions and velocities, and inspect their covariance spectrum. Near-zero eigenvalues indicate directions with little observed variation. Additional trajectories should introduce variation along those weakly covered directions.

We have expanded the pairing description and training-pair audit in Appendix E.1, L1177–1188. The synthetic correspondence-error and limited-coverage results are presented in Figure 14, and discussed in Appendix D.5, L1107–1109.

AuthorsOpenReview ↗

We sincerely appreciate your time and detailed feedback (3/4)

Q4. To what extent are exact whitening and global optimization satisfied in practice, and does training batch size affect block identifiability?

A4. Thank you for raising this point. We agree that the theoretical conditions and their practical approximations should be distinguished explicitly.

Whitening and global optimization in practice: Section 3.4, Eq. (11) states the exact normalization constraint and explains that finite-weight variance regularization does not itself impose exact whitening. Precise whitening and global optimality are theoretical conditions. In practice, they are approximated through limited sample normalization and numerical optimization. We provide the following diagnostics of the synthetic pipeline:

=\frac{\lVert\widehat{\operatorname{Cov}}(r)-I_d\rVert_F}{\sqrt d}, \qquad E_{\mathrm{iso}} =\frac{\lVert\widehat Q^\top\widehat Q-I_d\rVert_F}{\sqrt d}.$$ | Diagnostic | Ecov | R2 | Eiso | |---|---:|---:|---:| | Result | $0.0313 \pm 0.0038$ | $0.9977 \pm 0.0016$ | $0.0192 \pm 0.0020$ | These results indicate a close approximation to the predicted geometry under the default setting. *Covariance estimation:* As described in Appendix D.1, the original two-stage experiment freezes the encoder and fits centered ZCA whitening using 30,000 unlabeled representations before spectral recovery. The centered sample covariance uses the $N-1$ denominator, and the ZCA transform uses an eigenvalue floor of $10^{-5}$. *Batch size:* To examine training batch size separately, we train the end-to-end Eq. (10) model with batch sizes 64, 256, and 1,024, fixing the number of updates at 10,000. | Batch size | $R_H^2$ | $R_L^2$ | Purity | $\lVert A_{12}\rVert_F$ | $\lVert A_{21}\rVert_F$ | |---|---:|---:|---:|---:|---:| | 64 | $0.9980 \pm 0.0020$ | $0.9957 \pm 0.0049$ | $0.9992 \pm 0.0011$ | $0.0295 \pm 0.0254$ | $0.0540 \pm 0.0479$ | | 256 | $0.9979 \pm 0.0017$ | $0.9954 \pm 0.0046$ | $0.9992 \pm 0.0010$ | $0.0344 \pm 0.0230$ | $0.0543 \pm 0.0452$ | | 1,024 | $0.9979 \pm 0.0017$ | $0.9954 \pm 0.0047$ | $0.9992 \pm 0.0010$ | $0.0344 \pm 0.0237$ | $0.0530 \pm 0.0471$ | Block recovery remains similar across the tested batch sizes. Thus, under this setting and fixed update budget, recovery is stable over the tested range despite imperfect covariance normalization. We have added the normalization diagnostics and numerical results in **Appendix D.1, L893–906**, with the covariance-estimation procedure in **L873–883**, and **Figure 14**, presents the fixed-update batch-size comparison. --- **Q5.** When should the invariant representation improve downstream performance, and when can the variant block remain useful? **A5.** Thank you for this question. The preferred representation depends on whether $r_H$ and the policy’s other inputs are sufficient for the action, and whether the additional information in $r_L$ remains reliable at deployment. As discussed in Section 4.2 (L471–479) and Appendix E.2 (L1338–1344), neither representation interface dominates uniformly. Using $r_H$ alone can help when it retains the action-relevant information and deployment changes mainly affect signals in $r_L$. For example, a camera change may alter appearance without changing the information required for the control action. Under limited downstream supervision, restricting the input to $r_H$ can reduce reliance on these changing cues. The full representation can be preferable when the correct action also depends on execution information that is not sufficiently available in $r_H$ or the other policy inputs. For the same pushing goal, approaching the object from different directions may require different next actions. If $r_L$ provides useful execution information that remains reliable at deployment, retaining both blocks can help. A full-representation policy can, in principle, learn to ignore unreliable signals in $r_L$. The practical value of restricting its input to $r_H$ is to reduce reliance on potentially unstable signals when downstream data are limited. We have made these conditions explicit in the updated discussion of the two policy inputs. We have clarified the role of the two policy inputs in **Section 4.2, L477–479**, and expanded this discussion in **Appendix E.2, L1338–1344**, alongside the sample-efficiency results in **Figure 15**. The discussion relates the choice of input to task sufficiency and to what the policy learns from the available demonstrations.
Reviewer_Mhv3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Prior work on joint-embedding predictive architectures (JEPAs) provides reconstruction-free latent representations, but existing identifiability results do not establish the invariant–variant decomposition of the latent state. The authors propose SplitJEPA, which combines latent-space prediction with consistency across invariant correspondences to resolve the ambiguity between invariant and variant blocks without observation reconstruction. Under stationary Gaussian predictive dynamics and sufficient variation across invariant correspondences, the authors establish block identifiability up to independent within-block orthogonal transformations. Experiments on synthetic nonlinear systems and robotic manipulation tasks evaluate the proposed approach in terms of block recovery and downstream robustness.

Critical strengths
  1. The core idea is compelling and timely. While JEPA-based approaches have gained increasing attention, the identifiability of invariant and variant latent factors remains relatively underexplored. Given the fundamental role of invariance in representation learning, this work provides a valuable theoretical foundation for further research in this direction.

  2. The proposed identifiability result is theoretically convincing under the stated assumptions and conditions. The analysis provides a clear characterization of when invariant and variant latent subspaces can be identified without reconstruction.

  3. The experiments on synthetic data provide convincing support for the theoretical claims. The results demonstrate the effectiveness of the proposed approach in recovering the invariant–variant latent structure under controlled settings.

Critical weaknesses
  1. The full method does not consistently outperform its ablated variants. In particular, the diversity component shows task-dependent effects, improving performance in some settings while degrading it in others. This inconsistency raises questions about how the theoretical sufficient-variation condition translates into empirical performance, suggesting a potential gap between the theoretical analysis and practical implementation.
Questions for authors
  1. Some notation is confusing. For example, f~\tilde{f} in Theorem 1 is introduced without sufficient explanation, making it difficult to follow at first. Could the authors clarify the notation and consider revising it for readability?

  2. Clarifying how the proposed block-identification framework differs from prior equivariant self-supervised learning would help contextualize its contribution. Could the authors briefly explain the key differences, particularly in the learning objectives and theoretical guarantees?

5 comments
AuthorsOpenReview ↗

We sincerely appreciate your thoughtful and constructive comments (1/2)

Dear Reviewer Mhv3,

We sincerely appreciate your thoughtful and constructive comments. Below, we clarify the role of diversity, explain the notation, and discuss the connection to equivariant self-supervised learning.


W1. The diversity component has task-dependent effects. How does the sufficient-variation condition translate into practical performance?

Response. Thank you for raising this point. We agree that it is important to distinguish the sufficient condition for block identification from downstream performance. Theorem 2 establishes when the partition is identifiable; it does not assert that adding trajectory diversity must monotonically improve control performance.

As stated in Assumption 1, Eq. (7), sufficient variation requires

=\mathbb E_{\mathcal D_{\mathrm{inv}}} [\Delta z_L\Delta z_L^\top]\succ0$$ within correspondences that preserve $z_H$. Example 2 and Figure 4 in Section 3.2 explain why complementary changes can jointly cover the variant subspace. The discussion in L314–325 further emphasizes that the relevant diversity is measured within the specified correspondence. In practice, cross-trajectory pairing can introduce both broader variant coverage and less precise state alignment. As already discussed in Section 4.2 (L486–490) and Appendix E.1 (L1177–1188), trajectories are approximately aligned by execution progress. Additional executions can reveal previously uncovered variant directions, but paired frames can also differ in instantaneous physical state. When such differences affect the intended invariant content, they introduce correspondence error. We have clarified the distinction between variation coverage and correspondence precision in **Appendix E.1**. **Appendix D.5, L1107–1109, and Figure 14**, additionally examine incomplete variant coverage and inaccurate pairs, complementing the task-dependent diversity discussion in **L486–490 and Table 11**. The role of the two policy inputs is discussed in **L477–479** and **Appendix E.2, L1338–1344**; block identifiability does not imply a uniform downstream advantage for either input. --- **Q1.** Could the notation in Theorem 1 be clarified for readability? **A1.** Thank you for this helpful suggestion. We agree that more explicit definitions near the theorem would improve readability. The latent blocks and representation are introduced in Section 2, Eqs. (1)–(2). In Theorem 1, $\psi:\mathbb R^{d_L}\to\mathbb R^{d_H}$ is an arbitrary smooth nonlinear function, used to define $\widetilde z_H=z_H+\psi(z_L),\widetilde z_L=z_L.$ The alternative observation map is $\widetilde f(\widetilde z_H,\widetilde z_L)=f\bigl(\widetilde z_H-\psi(\widetilde z_L),\widetilde z_L\bigr).$ Here, the tilde denotes the alternative latent parameterization and its corresponding observation map. $J_\psi(z_L)$ is the Jacobian of $\psi$ at $z_L$. As explained in L162–167, its appearance in the off-diagonal block of the transformation Jacobian means that the alternative first block contains variant information whenever $J_\psi$ is nonzero. To make the notation easier to follow, we have added a concise notation table in **Appendix A, Table 3**.
AuthorsOpenReview ↗

We sincerely appreciate your thoughtful and constructive comments (2/2)

Q2. The difference between equivariant self-supervised learning and SplitJEPA.

A2. Thank you for this question. Equivariant self-supervised learning seeks representations whose responses to transformations follow corresponding latent actions. For example, Schmidt et al. [1] learn a latent group action and identify the latent space up to a global linear transformation (Theorem 1 in [1]).

Our focus is on identifying a designated invariant/variant partition using predictive relations and shared-factor correspondences. The objective combines predictive alignment, invariant-block consistency, and variance-preserving regularization, as given in Section 3.4, Eqs. (9)–(10). The corresponding guarantee is stated in Theorem 2. Under the predictive and normalization conditions, global recovery yields an orthogonal latent representation. Matched-pair consistency with sufficient variation removes variant-to-invariant mixing, and orthogonality removes the reverse mixing. Both blocks are consequently recovered up to independent within-block orthogonal transformations; the argument is detailed in Appendix C.2.

Thus, our contribution concerns how prediction, correspondence, and normalization jointly determine the specified partition, with its remaining ambiguity explicitly characterized within each block.

[1] Equivariance by Contrast: Identifiable Equivariant Embeddings from Unlabeled Finite Group Actions. NeurIPS 2025

We have added a concise comparison with equivariant self-supervised learning in the Introduction, and expanded this connection in Appendix B, L652–654.


Thank you again for your constructive feedback. We hope these clarifications help make the theoretical conditions and their practical interpretation clearer.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer Mhv3,

Thank you for your time and thoughtful feedback on our paper. We would greatly appreciate it if you could let us know whether our responses and clarifications adequately address your concerns. Thank you again for your consideration!

Best regards,

Authors of Submission 28460

Reviewer_fSrWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the invariant and variant parts of a latent state can be identified by a JEPA without reconstruction. Building on LeJEPA’s result that predictive alignment under whitening recovers the latent state up to a global orthogonal transformation, it shows (Theorem 2) that an invariance loss on the designated invariant block, over correspondences whose variant changes span the variant subspace (Assumption 1), forces that orthogonal transformation to be block-diagonal, so both blocks are identified up to within-block isometries; Theorem 3 notes that a predictor on the invariant block is then insensitive to variant perturbations. A synthetic Ornstein–Uhlenbeck study with post-hoc spectral recovery on a frozen LeJEPA encoder confirms the mechanism, and behavior-cloning experiments on ManiSkill PickCube and PushCube with a jointly trained ResNet-50 encoder show large gains in success under camera, lighting and combined visual shifts relative to raw pixels, R3M and LeJEPA.

Critical strengths

On the empirical side, the strongest result is PushCube under camera and combined visual shifts, where closed-loop success rises from 25–45% and 13–30% for the baselines to 72–75% and 65–70% with the learned representation, at unchanged in-distribution success and already with 10% of the demonstrations (Table 2, Figure 13); this is a practically meaningful robustness gain, though, as noted below, the paper’s own ablation attributes it to the invariance pairs rather than to the block split.

Critical weaknesses

My concern is that the paper’s claim, reconstruction-free identification of the invariant and variant blocks, is not what the experiments test. The theory itself is a small step: the closest prior work is the paper’s own foundation, since Theorem 2 adds three lines of linear algebra to Klindt et al. (2026), using the invariance argument of von Kügelgen et al. (2021) in its linear special case, under LeJEPA’s assumptions of isotropic Gaussian AR(1) latents, exact whitening and a global optimum that Table 4 shows is not reached at moderate observation nonlinearity (R2R^2 = 0.88 at coupling 2.0). A small theoretical step can still be valuable if the method is shown to work, but the synthetic experiment applies a post-hoc spectral rotation, computed from ground-truth-matched pairs, to a frozen LeJEPA encoder, so block purity near one follows from global linear recovery by construction, and the joint objective of Eq. 10 is never evaluated against ground truth. The robotic experiments in turn never test identifiability although the simulator exposes the true state, and their gains are attributed by the paper’s own ablation to the invariance term (PushCube camera OOD 71.5 →\rightarrow 19.7 without it) rather than to the block split, with no invariance-only full-representation baseline and a LeJEPA baseline that lies far below every ablation of the proposed model (PickCube ID 16.3 versus 39.7–46.3). What would make the paper convincing is an end-to-end synthetic evaluation of Eq. 10 including assumption violations, matched-information baselines, and linear probes of rHr_H and rLr_L against simulator ground truth (cube and goal positions versus camera pose and lighting).

Questions for authors

(1) Every ablation of your model, including those without the predictive term (39.7%) and without the invariance term (44.3%), reaches roughly three times LeJEPA’s PickCube ID success (16.3%). Since removing either distinguishing ingredient leaves the model far above the baseline, the difference must come from a factor the ablations never remove. Please state exactly how the LeJEPA baseline was trained (data, initialization, encoder, predictor, regularizer, budget) and what differs from your pipeline, and report a LeJEPA run that matches your pipeline in everything except the invariance loss and the split. Without this, the baseline comparison in Table 2 cannot be interpreted.

(2) Does an encoder trained from scratch with the joint objective of Eq. 10 recover the blocks in the synthetic setting, and how do block purity and the leakage terms depend on λinv\lambda_{\mathrm{inv}}, on the coupling strength of the observation map, and on departures from the model (non-Gaussian latents, coordinate-dependent ρ\rho)?

(3) Please report linear-probe R2R^2 from rHr_H and rLr_L to the simulator’s task-relevant state (cube and goal positions) and to the nuisance factors (camera azimuth, lighting intensity) on both tasks; this tests Theorem 2 where it matters, whereas Figure 14 measures what the invariance loss directly optimizes.

(4) Please add a control in which the same encoder is trained with the invariance loss on the full representation using the same pairs, without split or predictive term, so that the gain of the decomposition can be separated from the gain of seeing paired views.

(5) How do you position the invariant/variant split as a reconstruction-free goal against the following uncited work: Xiao et al., “What should not be contrastive in contrastive learning”, ICLR 2021; Yao et al., “Multi-view causal representation learning with partial observability”, ICLR 2024; Schmidt et al., “Equivariance by contrast: identifiable equivariant embeddings from unlabeled finite group actions”, arXiv 2510.21706?

8 comments
AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (1/4)

Dear Reviewer fSrW,

We sincerely appreciate your time and thoughtful feedback. Your suggestions are helpful for strengthening the empirical assessment of the identification mechanism. In response, we provide additional end-to-end synthetic experiments and ground-truth probes of the robotic representations. Please find our detailed point-by-point responses below.


Q1. Training protocol of the original LeJEPA baseline and performance comparison under matched training pipelines.

A1. We agree that the original comparison does not isolate the contribution of invariant consistency and decomposition from other pipeline differences. We clarify the original baseline below and provide the requested matched predictive control.

Original baseline and interpretation of Table 11. The LeJEPA baseline in Table 2 was trained on the same PickCube demonstration corpus, using an ImageNet-initialized ResNet-50 encoder with a single 128-dimensional projection. Training used temporal transitions without matched-pair invariant consistency. It was pretrained for 200 epochs with batch size 32 using AdamW and an initial learning rate of 3×10−43\times10^{-4}, after which the encoder was frozen for behavior cloning. In contrast, Table 11 is a within-SplitJEPA ablation: each variant retains the partitioned architecture and downstream pipeline while removing the indicated component. The 39.7%39.7\% result retains invariant-pair training but removes prediction; the 44.3%44.3\% result retains prediction but removes invariant consistency. Both use the rHr_H-only policy interface. Consequently, neither row is an unsplit predictive control matched to the complete pipeline.

Matched predictive control. Following your suggestion, we add an unsplit predictive model that uses the common training and downstream pipeline, retains the other shared training terms, and removes invariant consistency and the representation split. This provides a more direct comparison with Full SplitJEPA using [rH,rL][r_H,r_L].

TaskMethodIDCamera OODLighting OODCombined Visual OOD
PickCubeOriginal LeJEPA16.3±0.916.3 \pm 0.910.0±0.010.0 \pm 0.010.0±0.010.0 \pm 0.010.0±0.010.0 \pm 0.0
PickCubeUnsplit Prediction37.0±1.437.0 \pm 1.416.0±2.916.0 \pm 2.926.0±7.826.0 \pm 7.813.3±2.413.3 \pm 2.4
PickCubeFull SplitJEPA, [rH,rL][r_H,r_L]46.3±4.946.3 \pm 4.927.7±0.927.7 \pm 0.943.3±2.943.3 \pm 2.930.3±1.730.3 \pm 1.7
PushCubeOriginal LeJEPA67.3±20.167.3 \pm 20.125.0±2.625.0 \pm 2.631.0±31.231.0 \pm 31.228.7±17.528.7 \pm 17.5
PushCubeUnsplit Prediction94.3±4.594.3 \pm 4.534.0±1.734.0 \pm 1.784.7±11.084.7 \pm 11.022.2±4.922.2 \pm 4.9
PushCubeFull SplitJEPA, [rH,rL][r_H,r_L]99.3±0.699.3 \pm 0.675.3±8.175.3 \pm 8.198.3±1.598.3 \pm 1.570.2±12.970.2 \pm 12.9

The matched control substantially improves ID success over the original baseline. This confirms that pipeline differences contribute to the original performance gap. The improvement is not uniform across OOD conditions: PushCube combined visual success does not improve over the original baseline. Full SplitJEPA nevertheless retains additional gains over the matched predictive control, particularly under visual shifts. Thus, the complete method’s improvement persists after matching the pipeline.

We have clarified the original baseline and the Unsplit Prediction control in the updated manuscript, Appendix E.1, “Downstream Policies and Evaluation,” L1267–1273. The matched comparison is included in Table 2, and the within-model ablations appear in Table 11.

AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (2/4)

Q2. Block recovery from scratch using the joint objective of Eq. 10 in the synthetic setting. Effects of invariance weight, observation coupling, and latent dynamics misspecification on purity and leakage.

A2. Thank you for this suggestion. The original comparison uses a shared frozen encoder and spectral recovery. We now train a randomly initialized MLP with Eq. (10) and evaluate its designated blocks directly, without spectral rotation. This experiment uses a different invertible observation generator and a minibatch mean/covariance penalty, so we report it separately from the original inverse-coupling/SIGReg setting. Results are mean ±\pm standard deviation over five seeds.

Training conditionGlobal R2R^2RH2R_H^2RL2R_L^2Purity∥A12∥F\lVert A_{12}\rVert_F∥A21∥F\lVert A_{21}\rVert_F
λinv=0\lambda_{\mathrm{inv}}=00.9977±0.00160.9977 \pm 0.00160.9608±0.03330.9608 \pm 0.03330.9616±0.03280.9616 \pm 0.03280.9634±0.03190.9634 \pm 0.03190.3317±0.21440.3317 \pm 0.21440.3277±0.21960.3277 \pm 0.2196
λinv=0.1\lambda_{\mathrm{inv}}=0.10.9986±0.00130.9986 \pm 0.00130.9957±0.00410.9957 \pm 0.00410.9962±0.00350.9962 \pm 0.00350.9973±0.00250.9973 \pm 0.00250.0860±0.07410.0860 \pm 0.07410.0787±0.06430.0787 \pm 0.0643
λinv=1\lambda_{\mathrm{inv}}=10.9974±0.00230.9974 \pm 0.00230.9979±0.00170.9979 \pm 0.00170.9954±0.00460.9954 \pm 0.00460.9992±0.00100.9992 \pm 0.00100.0344±0.02300.0344 \pm 0.02300.0543±0.04520.0543 \pm 0.0452
λinv=10\lambda_{\mathrm{inv}}=100.9963±0.00160.9963 \pm 0.00160.9982±0.00040.9982 \pm 0.00040.9920±0.00540.9920 \pm 0.00540.9985±0.00200.9985 \pm 0.00200.0133±0.00480.0133 \pm 0.00480.0930±0.06330.0930 \pm 0.0633
Shuffled invariant pairs0.9785±0.00530.9785 \pm 0.00530.7743±0.06600.7743 \pm 0.06600.7645±0.06500.7645 \pm 0.06500.7845±0.07080.7845 \pm 0.07080.6269±0.10780.6269 \pm 0.10781.2964±0.21881.2964 \pm 0.2188

Joint training recovers both designated blocks. At λinv=1\lambda_{\mathrm{inv}}=1, purity reaches 0.9992±0.00100.9992\pm0.0010. Prediction alone preserves global recovery but leaves greater leakage; shuffled pairs disrupt separation. Increasing λinv\lambda_{\mathrm{inv}} reduces ∥A12∥F\lVert A_{12}\rVert_F, but the larger reverse leakage at weight 10 shows that stronger consistency does not uniformly improve recovery.

We also vary observation coupling, non-Gaussianity and coordinate-dependent correlations:

SweepConditionPurity∥A12∥F\lVert A_{12}\rVert_F∥A21∥F\lVert A_{21}\rVert_FGlobal R2R^2
Defaults=0.5s=0.5, Gaussian, ρi=0.5\rho_i=0.50.9992±0.00100.9992 \pm 0.00100.0344±0.02300.0344 \pm 0.02300.0543±0.04520.0543 \pm 0.04520.9974±0.00230.9974 \pm 0.0023
Observation couplings=0.251.0000±0.00001.0000 \pm 0.00000.0034±0.00100.0034 \pm 0.00100.0105±0.00220.0105 \pm 0.00220.9999±0.00010.9999 \pm 0.0001
Observation couplings=10.9901±0.01000.9901 \pm 0.01000.1410±0.04010.1410 \pm 0.04010.2137±0.14020.2137 \pm 0.14020.9692±0.01100.9692 \pm 0.0110
Observation couplings=20.9685±0.02500.9685 \pm 0.02500.2306±0.06770.2306 \pm 0.06770.4235±0.21810.4235 \pm 0.21810.8894±0.02670.8894 \pm 0.0267
Non-Gaussianityτ=0.25\tau=0.250.9993±0.00100.9993 \pm 0.00100.0336±0.02320.0336 \pm 0.02320.0519±0.04470.0519 \pm 0.04470.9974±0.00230.9974 \pm 0.0023
Non-Gaussianityτ=0.5\tau=0.50.9995±0.00050.9995 \pm 0.00050.0335±0.02190.0335 \pm 0.02190.0446±0.02850.0446 \pm 0.02850.9975±0.00190.9975 \pm 0.0019
Non-Gaussianityτ=1\tau=10.9996±0.00040.9996 \pm 0.00040.0333±0.01600.0333 \pm 0.01600.0429±0.02110.0429 \pm 0.02110.9976±0.00150.9976 \pm 0.0015
Coordinate-dependent ρi\rho_iRange [0.4,0.6]0.9989±0.00090.9989 \pm 0.00090.0430±0.01510.0430 \pm 0.01510.0776±0.03210.0776 \pm 0.03210.9962±0.00130.9962 \pm 0.0013
Coordinate-dependent ρi\rho_iRange [0.3,0.7]0.9981±0.00160.9981 \pm 0.00160.0567±0.01700.0567 \pm 0.01700.1016±0.04670.1016 \pm 0.04670.9956±0.00200.9956 \pm 0.0020
Coordinate-dependent ρi\rho_iRange [0.1,0.9]0.9808±0.00810.9808 \pm 0.00810.1357±0.02300.1357 \pm 0.02300.3774±0.08030.3774 \pm 0.08030.9867±0.00140.9867 \pm 0.0014

Stronger coupling reduces global recovery and purity while increasing leakage under the fixed optimization budget. Recovery remains stable across the tested non-Gaussian conditions and moderate correlation heterogeneity; greater heterogeneity degrades separation. These results provide empirical sensitivity evidence beyond the exact predictive model.

We have added “Joint Training” in Appendix D.5, with the core comparison in Table 6, and the sensitivity sweeps in Figure 13.

AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback (3/4)

Q3. The ground-truth linear probes for task-relevant states and nuisance factors on both robotic tasks.

A3. The original analysis in Appendix E.2, Figure 16 measures differential responses to matched viewpoint and trajectory changes. We therefore supplement it with linear probes of simulator ground truth on both tasks. We freeze the encoder and render each state under crossed conditions comprising three camera azimuths and three lighting levels. We fit linear ridge regressions using disjoint matched-task groups for training, validation, and testing, with regularization selected on the validation groups. Position targets are three-dimensional for PickCube and planar for PushCube.

TaskMetricCubeGoalCameraLighting
PickCuberHr_H0.9160±0.01070.9160 \pm 0.01070.1423±0.00860.1423 \pm 0.00860.1129±0.01370.1129 \pm 0.01370.0379±0.00440.0379 \pm 0.0044
PickCuberLr_L0.8350±0.08200.8350 \pm 0.08200.1501±0.02360.1501 \pm 0.02360.5254±0.00700.5254 \pm 0.00700.0624±0.00770.0624 \pm 0.0077
PickCube[rH,rL][r_H,r_L]0.9317±0.01440.9317 \pm 0.01440.1699±0.01380.1699 \pm 0.01380.5716±0.04250.5716 \pm 0.04250.0818±0.01400.0818 \pm 0.0140
PushCuberHr_H0.6582±0.04190.6582 \pm 0.04190.6105±0.05480.6105 \pm 0.05480.0145±0.06710.0145 \pm 0.06710.0572±0.03040.0572 \pm 0.0304
PushCuberLr_L0.6575±0.02250.6575 \pm 0.02250.6029±0.02250.6029 \pm 0.02250.0268±0.02010.0268 \pm 0.02010.0993±0.00890.0993 \pm 0.0089
PushCube[rH,rL][r_H,r_L]0.7797±0.01780.7797 \pm 0.01780.7486±0.02810.7486 \pm 0.02810.0705±0.10770.0705 \pm 0.10770.1788±0.01800.1788 \pm 0.0180

PickCube exhibits the clearest task-state/viewpoint specialization. Cube position is more accurately decoded from rHr_H, while camera azimuth is substantially more accurately decoded from rLr_L. Thus, the lower viewpoint sensitivity of rHr_H is accompanied by strong retention of cube-position information, and rLr_L retains more linearly accessible viewpoint information. With physical state and lighting fixed, signed camera-angle differences are also more accurately decoded from rLr_L. On PushCube, both blocks retain substantial task-state information. Concatenating them improves cube-position decoding and goal-position decoding. Their differentiation is more apparent under controlled viewpoint changes: signed camera-angle differences are more accurately decoded from rLr_L than from rHr_H.

These probes extend Figure 16 by examining both information retention and factor selectivity against simulator ground truth. They provide an empirical assessment of the intended specialization, with clearer task-state/viewpoint differentiation on PickCube and complementary task-state information on PushCube.

We have added the probe protocol and its interpretation in Appendix E.2, “Ground-Truth Probes”, with results in Table 10. These additions complement the sensitivity analysis in Figure 16.


Q4. Please add a control trained with invariant consistency on the full representation using the same pairs, without a split or predictive term, to separate the gain of decomposition from the gain of paired views.

A4. We add the requested unsplit full-representation invariance control, using the same encoder and invariant pairs, with prediction disabled.

For a controlled comparison, we use the original w/o Predictive (rHr_H+rLr_L) variant in Table 11 as the reference. Both models have the same total representation dimension, disable the predictive loss, and retain the other training terms. They differ in where invariant consistency is imposed:

  • Split, without prediction: invariant consistency is applied only to rHr_H, while the downstream policy receives both rHr_H and rLr_L.
  • Unsplit, without prediction: invariant consistency is applied to the entire representation, which is passed to the downstream policy.
TaskMethodIDCamera OODLighting OODCombined Visual OOD
PickCubeFull-space Invariance42.3±1.242.3 \pm 1.218.7±0.518.7 \pm 0.541.7±3.741.7 \pm 3.722.0±0.822.0 \pm 0.8
PushCubeFull-space Invariance99.0±1.099.0 \pm 1.065.2±11.665.2 \pm 11.6100.0±0.0100.0 \pm 0.062.5±5.162.5 \pm 5.1

Full-space Invariance performs similarly to or better than the split counterpart without prediction in several conditions. Thus, paired-view consistency is an important source of the observed control gains. This distinction complements the within-model ablations discussed in Section 4.2 (L480–490).

We have included this control as Full-space Invariance in Table 11. Its training configuration is specified in L1271–1273, so the two comparisons can be read together.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.