acceptodds
Under review as a conference paper at ICLR 2027

DSReg: Provably Recovering Individual World Latents without Reconstruction

Abstract

Methods that recover individual latent variables of the world, from nonlinear ICA to dictionary learning and causal representation learning, anchor the latents to observations through reconstruction, auxiliary supervision, or distributional asymmetries such as non-Gaussianity. Methods without these anchors, including joint-embedding predictive architectures (JEPAs), identify the latent state only up to a linear transformation, so individual latents remain mixed. We close this gap: individual world latents can be provably recovered with no reconstruction, no decoder, and no labels. The key condition is Structural Diversity: different latents leave distinct dependency footprints on observations, just as no two snowflakes are alike. Building on the linear identifiability that LeJEPA provides, we prove that under Structural Diversity and a standard faithfulness condition, DSReg (Dependency-Sparsity Regularization) recovers individual world latents up to signed permutation. It applies post hoc to any linearly identified representation, reusing trained checkpoints at no loss over joint training, and establishes the first fully identifiable JEPA that recovers every world latent. Moreover, as a condition on dependency footprints, Structural Diversity is strictly weaker than all structural conditions of prior identifiable latent variable models. Across synthetic regimes, world model probes, learned visual encoders, and external renderers, DSReg preserves dense prediction while improving individual-latent recovery and downstream use with scales.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation learning, world model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ey2vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DSReg, a post-hoc method that resolves the rotational ambiguity in linearly identifiable representations by seeking a coordinate system with sparse observation-to-latent Jacobian structure. Under a Gaussian JEPA identifiability premise, distinct dependency footprints, and a functional no-cancellation condition, the authors prove that individual latent factors can be recovered up to sign and permutation. Experiments on synthetic environments, learned visual encoders, and rendered datasets report improved factor-wise recovery and performance on several tasks requiring sparse access to latent variables.

Critical strengths

The paper tackles an important limitation of predictive representations: retaining the complete latent state does not ensure that each coordinate captures a distinct factor of the world. It clearly distinguishes preservation of the overall information from recovery of individual factors. The theoretical analysis formalises the role and limits of dependency structure, while the appendix provides useful ablations studies.

Critical weaknesses
  1. The theoretical guarantees do not directly apply to the estimator used in the experiments. Theorem 2 concerns minimising the exact population support count, while Theorem 3 analyses a thresholded population criterion. In practice, the authors optimise an anchor-averaged ℓ1\ell_1 surrogate based on finite-sample local ridge estimates of the Jacobian. These objectives are not equivalent. Indeed, the paper reports cases in which the ℓ1\ell_1 objective has a mixed global minimiser even though the exact support criterion identifies the latents. Since exact-count refinement is demonstrated only on small benchmarks, the paper should establish when the practical estimator recovers the theoretically identified solution and report its failure rate and performance relative to the exact criterion.

  2. The external-renderer experiments may violate the theorem’s differentiability assumptions. In the 3D Shapes and dSprites experiments, continuous Gaussian latents are mapped onto finite factor grids before rendering. This makes the latent-to-image map piecewise constant, with a Jacobian that is zero almost everywhere. Under the paper’s definition, all dependency footprints would therefore be empty, violating structural diversity. The estimator appears instead to capture finite differences across grid levels. This may provide a useful empirical signal, but it is not the Jacobian support analysed by the theorem. The authors should resolve this mismatch, either by using a continuous generative map or by developing a criterion and analysis for discrete, piecewise-constant observations.

  3. The identifiability guarantee relies on a restrictive representation-learning premise. The theorem assumes that the learned representation is exactly an orthogonal transformation of the true Gaussian latent state, under a stationary Gaussian transition and at the population alignment optimum. The learned-encoder experiments approximate this idealised setting, but the robustness analysis considers Jacobian estimation error rather than deviations from linear identifiability. The authors should clarify the limits of the guarantee and characterise how representation error affects recovery before extending the claim to learned encoders.

  4. The evaluation lacks comparisons with closely related Jacobian-based methods. The paper discusses Independent Mechanism Analysis (IMA), which can use the same estimated local Jacobians but leaves joint diagonalisation and related baselines to future work. Because DSReg’s main contribution is to use observation-side dependency structure to resolve rotational ambiguity, comparisons with these approaches are important, particularly when their assumptions overlap. The existing comparisons with PCA, Varimax, FastICA, and sparse ICA do not establish whether DSReg outperforms alternative methods that exploit the same observation-side information.

  5. The empirical scope is limited relative to the paper’s world-model framing. The learned visual benchmarks use controlled synthetic scenes, and the external-renderer benchmarks use discretised factors. The paper does not evaluate DSReg on real video or physical systems, where noise, occlusion, and changing scene structure could affect local Jacobian estimates. Although the authors acknowledge this limitation, claims about recovering real-world variables should be tempered until validated in more realistic settings.

Questions for authors

Please see the weakness section.

9 comments
AuthorsOpenReview ↗

We truly appreciate your insightful and constructive suggestions (1/5)

We truly appreciate your insightful and constructive suggestions. We are glad that you find the problem important, the distinction between preserving information and recovering individual factors clear, and the theoretical analysis and ablations useful. In light of your suggestions, we have added new theoretical results, new experiments, and detailed discussions. Please find our detailed responses as follows.


Q1. The guarantees do not directly apply to the estimator. Theorems 2 and 3 concern the exact support count and a thresholded criterion, while in practice the authors optimise an anchor-averaged ℓ1\ell_1 surrogate on finite-sample ridge estimates, and the ℓ1\ell_1 objective can have a mixed global minimiser. The paper should establish when the practical estimator recovers the identified solution and report its failure rate and performance relative to the exact criterion.

A1. Thanks a lot for raising this. Accordingly, we have proved new results for Equation 4 itself, including under finite-sample ridge estimates (Appendix A.5, L1445), and we now report how the estimator fails relative to the exact criterion. In short, under a sufficient condition the minimizers of Equation 4 are exactly the signed permutations and stay close to them under small Jacobian error, and in experiments with exact Jacobians a support count started from the ℓ1\ell_1 solution returns the true axes in almost every world.

In the original version, we mainly reported the anchor-averaged ℓ1\ell_1 objective since ℓ1\ell_1 is the standard relaxation of ℓ0\ell_0, and the local form is mainly for scalability to large models. This is also the case in previous sparsity-based identifiability results, which state the theorem through the support and estimate with ℓ1\ell_1-type surrogates. For instance, [4] uses an ℓ1\ell_1 norm on the Jacobian as an approximation of ℓ0\ell_0 during estimation, and [5] states its theorem through ℓ0\ell_0 and estimates with continuous surrogates such as ℓ1\ell_1 and SCAD.

For the theory, let D(a)D(a) be the matrix of local slopes at anchor aa, under any measure on anchors, so the statements cover the empirical measure on the mm anchors. Let wiw_i be the mass of latent ii on the (row, anchor) pairs that it alone drives, let mim_i be its mass on pairs shared with another latent, and let c=E∑r(∥Dr∥1−∥Dr∥2)c=\mathbb{E}\sum_r(\Vert D_r\Vert_1-\Vert D_r\Vert_2).

Proposition 4. If wi>(1+2)miw_i > (1+\sqrt{2})m_i and wi>(3+22)cw_i > (3+2\sqrt{2})c for every ii, then the minimizers of the ℓ1\ell_1 objective over O(d)O(d) are exactly the signed permutations.

Corollary 1. Let D∘D^\circ satisfy this condition, with margins γ=min⁡i((2−1)wi−mi)\gamma=\min_i((\sqrt{2}-1)w_i-m_i) and ξ=(3−22)min⁡iwi−c\xi=(3-2\sqrt{2})\min_i w_i-c, and let D^=D∘+E\widehat D=D^\circ+E with εˉ=E∑r∥Er∥2\bar\varepsilon=\mathbb{E}\sum_r\Vert E_r\Vert_2. If 2dεˉ<ξ2\sqrt{d}\bar\varepsilon < \xi, then every minimizer of Equation 4 on D^\widehat D lies within 2dεˉ/γ2\sqrt{d}\bar\varepsilon/\gamma of a signed permutation, measured as the sum of row-wise ℓ1\ell_1 distances.

Proposition 5. A signed permutation is a strict local minimizer if a first-order inequality holds in every skew direction, and it is not a local minimizer if the inequality is reversed in one direction.

The error EE enters only via its mean across anchors. Therefore, it can contain ridge error, finite-sample error and representation error simultaneously. The condition is both sufficient and far from being necessary. In the random worlds where each latent has its own private rows and shared rows are added with increasing strength, the ℓ1\ell_1 search found a signed permutation every time, even for the worlds that lie outside this condition.

[4] Zheng et al., "On the Identifiability of Nonlinear ICA: Sparsity and Beyond", NeurIPS 2022.

[5] Ng et al., "On the Identifiability of Sparse ICA without Assuming Non-Gaussianity", NeurIPS 2023.

AuthorsOpenReview ↗

We truly appreciate your insightful and constructive suggestions (2/5)

For the experiments, we now separate a criterion failure, where a mixed rotation has a lower objective than the true axes, from an optimizer failure, where the true axes win but are not reached (L346-348, L2189-2191). On the protocol of the submission with exact Jacobians, the results (latent MCC, N=16N=16) are as follows. The first row is the representation before any rotation is fitted, and the two count rows search, starting from the ℓ1\ell_1 solution, for the rotation with the fewest nonzero Jacobian entries, counted at each anchor or once over all anchors.

CriterionDiverseMinimalNested
No rotation0.506 ± 0.0280.506 ± 0.0280.506 ± 0.028
ℓ1\ell_1, Equation 41.000 ± 0.0000.966 ± 0.0361.000 ± 0.000
Count at each anchor, started from ℓ1\ell_11.000 ± 0.0001.000 ± 0.0001.000 ± 0.000
Global count, started from ℓ1\ell_11.000 ± 0.0001.000 ± 0.0001.000 ± 0.000

Started from the ℓ1\ell_1 solution, the global count, a thresholded form of the support count in Theorem 2, returns the true axes and the true count in every world at N=8N=8 and N=16N=16. Where derivative magnitudes make the ℓ1\ell_1 optimum a mixed rotation, as in gα(z)=(z1+αz2+0.1sin⁡z2,z2)g_\alpha(z)=(z_1+\alpha z_2+0.1\sin z_2,z_2) at α≥0.5\alpha \ge 0.5, the same refinement returns the true axes in almost every run and otherwise keeps the ℓ1\ell_1 solution.

We have also extended this protocol up to N=128N=128. Across the 600 worlds from N=8N=8 to N=128N=128, the ℓ1\ell_1 solution is more than 10 degree away from the true axes in 280 cases, which result from a criterion failure (167 cases) or an optimizer failure (113 cases), mostly at N≥64N \ge 64, and its MCC remains 0.92 or above at N=128N=128. Started from the ℓ1\ell_1 solution after local polishing, the count at each anchor returns the true axes in 594 of the 600 worlds and has an MCC of 0.998 or above in the other six. The results at N=128N=128 are as follows.

Criterion, N=128N=128DiverseMinimalNested
No rotation0.243 ± 0.0030.243 ± 0.0030.243 ± 0.003
ℓ1\ell_1, Equation 40.998 ± 0.0010.947 ± 0.0110.961 ± 0.007
Count at each anchor, started from ℓ1\ell_11.000 ± 0.0001.000 ± 0.0001.000 ± 0.000

With Jacobians estimated by local ridge regression, the in-sample objective often prefers a mixed rotation, but on held-out anchors the true axes win in almost every such world, so nearly all of these failures come from estimation error on the fitted anchors. With a label-free threshold set from the standard error of the ridge slopes, the refinement raises the diverse family at N=128N=128 from 0.876 to 0.999.

As a label-free diagnostic, the distance the refinement moves the ℓ1\ell_1 solution flags its failures with AUROC 0.99, for exact and estimated Jacobians alike. On the learned benchmarks, where the refinement is kept only if it lowers the count on held-out anchors, it almost never does, so the ℓ1\ell_1 solution stands, and Equation 4 there is already within 0.005 of the supervised oracle on 3DShapes and both visual suites.

AuthorsOpenReview ↗

We truly appreciate your insightful and constructive suggestions (3/5)

Q2. The external-renderer experiments may violate the differentiability assumptions. Continuous latents are mapped onto finite factor grids, so the latent-to-image map is piecewise constant, with a Jacobian that is zero almost everywhere. The authors should resolve this mismatch, either with a continuous generative map or with an analysis for discrete observations.

A2. Thanks so much for this insightful point. In light of it, we have taken both routes you suggested, an analysis for grid observations and continuous renderers (Appendix A.6, L1611, and Appendix C.9, L2667). In short, on a grid the local regression estimates slopes with the same footprints as the grid, so our stability results apply to them, and on a continuous renderer of 3DShapes DSReg recovers the factors better than on the published grid.

For the analysis, on a grid the derivative ∂x/∂z\partial x/\partial z is zero almost everywhere, and the estimator reads finite differences across levels. For a local regression with Gaussian weights on a continuous regressor, Stein's identity shows that the regression estimates the Jacobian of the smoothed map; the smoothed map is the original map gg convolved with the kernel φϵ\varphi_\epsilon (that is, gϵ=g∗φϵg_\epsilon=g\ast\varphi_\epsilon), and this smoothed map is real analytic. The derivative of row rr in factor ii at each anchor is a Gaussian-weighted sum of the jumps of coordinate rr across each level boundary of factor ii. Hence, the derivative of row rr in factor ii is identically zero exactly when row rr does not change across any level of factor ii. Therefore, the smoothed map has the same footprints as the grid. An encoder of grid images is constant on each cell of the grid. Given a representation that is quantized along each factor, the population slope is Sri=Cov(gr,qi)/Var(qi)S_{ri}=\mathrm{Cov}(g_r,q_i)/\mathrm{Var}(q_i), which is precisely equal to zero outside the footprints. Propositions 4 and 5 and Corollary 1 all hold for any slope field and thus apply directly to both fields. What distinguishes the estimated slopes from those in these two fields (e.g., the neighborhood shape, the ridge penalty, or finite samples) enters Corollary 1 as error.

For the experiments, we replaced the grid lookup with differentiable renderers that keep the structure of both benchmarks. One port keeps a single sprite with shape, scale, orientation, and two positions, and the other keeps a single object in a room with three hues, scale, shape, and orientation. Two split worlds spread the same factors over separate objects as positive controls. Quantizing each factor to LL levels sweeps from coarse grids through the published grid sizes to the continuous map. The results (DSReg MCC) are as follows.

Levels per factor481632Published gridContinuous
3DShapes port0.8800.9170.9390.9460.9170.947
Split sprites0.9200.9490.9560.9630.9200.970
Split scene0.8970.9100.9320.9280.9140.915

On the 3DShapes port, recovery is highest for the continuous map, where DSReg reaches 0.947 against 0.667 for LeJEPA, and the published grid scores below it, so the grid does not inflate the result. DSReg improves over LeJEPA at every level on all four worlds, including the dSprites port. Scored against the true factors, rotations fitted to local ridge Jacobians also stay within 0.03 of rotations fitted to autograd Jacobians of the continuous renderers on all four worlds. In the manuscript, the external renderers are now presented together with this analysis, as empirical tests beyond the exact conditions of Theorem 2.

Reviewer_TZFxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper suggests Dependency-Sparsity Regularization (DSReg), which is a post-hoc method for recovering individual generative factors from representations identifiable up to an orthogonal transformation. It builds upon (or assumes) a linear-identifiability result for LeJEPA under Gaussian latent dynamics. Under an introduced Structural Diversity, which requires different latent variables to have distinct observation-dependency footprints, plus a functional non-cancellation, the paper shows that minimizing the global support of the observation Jacobian recovers individual factors up to signed permutation. They also establish approximate recovery for a thresholded population support criterion under bounded Jacobian perturbations. Practically, this reduces to estimating local observation-direction Jacobians from frozen representations through ridge regression and optimizes an orthogonal rotation using an averaged l1-dependency penalty, without global decoder training. Experiments cover synthetic footprint structures, learned visual encoders, external rendered datasets, and sparse downstream tasks. They demonstrate improvements in component recovery and few-shot access compared to native approach (as well as some classical post-hoc methods) while preserving dense predictive information.

Critical strengths
  1. Clean identifiability result for selecting coordinates within an orthogonal equivalence class. The paper isolates a meaningful problem left unresolved by linear representation identifiability. It also provides an elegant support-based argument for identifying the original coordinates. The robustness result for thresholded population support also strengthens the theoretical contribution.

  2. Strong controlled empirical results and a computationally practical post-hoc procedure. The experiments demonstrate substantial gains in component-wise recovery, with a large increase in latent MCC. The paper also contains extensive ablations on footprint structure, sparsity surrogates, Jacobian perturbations, training schedules, and computational scaling. Particularly valuable are the hard-support refinements and the comparison with a sparse-ICA criterion using the same estimated Jacobians.

Critical weaknesses
  1. The main recovery guarantees do not apply to the implemented objective. The central theorems concern the exact global functional-support count (Theorem 2) and a thresholded population support criterion (Theorem 3). In contrast, every experiment uses an averaged l1 relaxation (Equation 4).

  2. The novelty relative to closely related structural-identifiability and post-hoc methods is not sufficiently established. [1] already establish identifiable post-hoc recovery from frozen, linearly mixed representations using functional properties of image generation, including Gaussian or dependent factors (their assumptions and use of encoder Jacobians differ, so the practicality and empirical performance should be evaluated in the experiments). Likewise, [2] explicitly use generative-Jacobian (\ell_1) regularization to select latent orientations, which seems to provide at least a similar optimisation objective (even when not proving exact-recovery theorem). Also, it is unclear if recent work [3] is potentially sufficient for DSReg's component-wise result. PCA, FastICA, and Varimax applied only to latent samples cannot exploit the observation-dependency signal by construction. Although the adapted sparse-ICA baseline is useful, comparisons with closely related post-hoc structural methods would make the empirical contribution more convincing.

Refs: [1] Leemann, Tobias, et al. "When are post-hoc conceptual explanations identifiable?." Uncertainty in Artificial Intelligence. PMLR, 2023. [2] Rhodes, Travers, and Daniel Lee. "Local Disentanglement in Variational Auto-Encoders Using Jacobian $ L_1 $ Regularization." Advances in Neural Information Processing Systems 34 (2021): 22708-22719. [3] Zheng, Yujia, et al. "Diverse Dictionary Learning." arXiv preprint arXiv:2604.17568 (2026).

Questions for authors
  1. Practical recovery versus ideal support identifiability. Can the authors establish sufficient conditions under which the population l1 objective in Equation 4 has the signed-permutation minimizers identified by Theorem 2? How does the refinement perform systematically in settings where derivative magnitudes make the population l1 optimum a mixed rotation?

  2. Precise relationship to prior structural-identifiability results. Could the authors provide a theorem-level comparison with [1] and [3]? Does DSReg establish a genuinely different or stronger result compared particularly with [3], or can Theorem 2 be derived by specializing the existing framework to an orthogonal equivalence class?

8 comments
AuthorsOpenReview ↗

We sincerely thank you for your time and thoughtful review (1/4)

We sincerely thank you for your time and thoughtful review. We are glad that you find the identifiability result clean, the support-based argument elegant, and the empirical results strong with a computationally practical procedure. In response to your insightful suggestions, we have proved new theoretical results for the implemented objective and conducted new experiments. Please see below for our point-by-point responses.


Q1. The main recovery guarantees do not apply to the implemented objective. Theorems 2 and 3 concern the exact global support count and a thresholded population criterion, while every experiment uses the averaged ℓ1\ell_1 relaxation in Equation 4.

A1. Thanks for the suggestion. In light of it, we have added new results on both the theory and the experiments (Appendices A.5 and C.3). In the original version, we mainly reported the averaged ℓ1\ell_1 relaxation since ℓ1\ell_1 is the standard relaxation of ℓ0\ell_0, and the local approximation is mainly for scalability to large models. This is also the case in previous sparsity-based identifiability results. For instance, [4] states the theorem through the support and uses an ℓ1\ell_1 norm on the Jacobian as an approximation of ℓ0\ell_0 during estimation, and [5] states its theorem through ℓ0\ell_0 and estimates with continuous surrogates such as ℓ1\ell_1 and SCAD.

On the theory side, we now prove recovery for Equation 4 itself under a sufficient condition, including under estimated Jacobians (see A3 below).

On the experiment side, we compared Equation 4 with the support counts of the theorems and with global group ℓ1\ell_1 on the surrogate protocol of the submission, with exact Jacobians. Equation 4 already recovers the true axes or comes close to them, and a hard support count started from its solution closes the remaining gap. The results (latent MCC, N=16N=16) are as follows, where "No rotation" is the representation before any rotation is fitted.

CriterionDiverseMinimalNested
No rotation0.506 ± 0.0280.506 ± 0.0280.506 ± 0.028
ℓ1\ell_1, Equation 41.000 ± 0.0000.966 ± 0.0361.000 ± 0.000
Global group ℓ1\ell_11.000 ± 0.0000.913 ± 0.0331.000 ± 0.000
Count at each anchor, started from ℓ1\ell_11.000 ± 0.0001.000 ± 0.0001.000 ± 0.000
Global count, started from ℓ1\ell_11.000 ± 0.0001.000 ± 0.0001.000 ± 0.000
Global count, cold start0.862 ± 0.1180.774 ± 0.1120.830 ± 0.126

The global count is the thresholded version of the criterion of Theorem 2. In all worlds, at both N=8N=8 and N=16N=16, whenever it is initialized with the ℓ1\ell_1 solution, it gives the true axes and the true count. It stalls at N=16N=16 when started cold; therefore, it is the ℓ1\ell_1 objective that brings it to the solution. On this protocol, global group ℓ1\ell_1 (the direct convex relaxation of the count) does no better than Equation 4; thus, there is nothing lost by averaging over anchors.

This also holds up to N=128N=128. When initialized using the polished ℓ1\ell_1 solution, the count at each anchor produces the true axes in almost every world from N=8N=8 through N=128N=128, and achieves an MCC of 0.998 or above in the remaining ones. Results at N=128N=128 are as follows.

Criterion, N=128N=128DiverseMinimalNested
No rotation0.243 ± 0.0030.243 ± 0.0030.243 ± 0.003
ℓ1\ell_1, Equation 40.998 ± 0.0010.947 ± 0.0110.961 ± 0.007
Count at each anchor, started from polished ℓ1\ell_11.000 ± 0.0001.000 ± 0.0001.000 ± 0.000

On the learned benchmarks, Equation 4 is already within 0.005 of the supervised oracle on 3DShapes and on both visual suites.

[4] Zheng et al., "On the Identifiability of Nonlinear ICA: Sparsity and Beyond", NeurIPS 2022.

[5] Ng et al., "On the Identifiability of Sparse ICA without Assuming Non-Gaussianity", NeurIPS 2023.

AuthorsOpenReview ↗

We sincerely thank you for your time and thoughtful review (2/4)

A2. Thanks so much for the suggestion. The new comparison shows that DSReg has a larger mean than the other methods that read Jacobians on 3DShapes, and DSReg also reaches the supervised oracle in a world where the footprints overlap (where both joint diagonalization and IMA contrast fail). All three works have been discussed in the updated manuscript as part of the discussion of related work (Appendix B, L1822). A theorem-level comparison of DSReg with [1] and [3] is given in A4 below. For [2], the VAE decoder is shaped by a Jacobian ℓ1\ell_1 term during training, and that term comes without a recovery theorem. DSReg can be applied to a frozen representation with no global decoder. Its contribution is the identification result inside the orthogonal class, and we now also give sufficient conditions for the ℓ1\ell_1 objective itself.

In light of your suggestion, we have also added new experiments (Table 17, L2828; Table 18, L2862). The criteria on BaB_a are applied to the same anchor Jacobians of the same encoder; [1] uses the Jacobian of this encoder, and both generative baselines ([2] and an autoencoder with ℓ1\ell_1 sparsity following [3]) learn from pixels. Each method is evaluated on the same held-out data. The second column represents a rendered world in which neighbouring footprints overlap. The "No rotation" entry is what the encoder produces before any rotation is fitted. The results, in terms of latent MCC, are provided below.

Criterion3DShapesOverlapping footprints
No rotation0.644 ± 0.0530.600 ± 0.035
Varimax on latents0.507 ± 0.0150.558 ± 0.044
Column support on BaB_a0.763 ± 0.0570.952 ± 0.001
Varimax on BaB_a0.901 ± 0.0240.943 ± 0.002
DSReg0.922 ± 0.0050.951 ± 0.001
Joint diagonalization of Ba⊤BaB_a^\top B_a0.905 ± 0.0150.536 ± 0.011
IMA contrast over rotations0.905 ± 0.0150.545 ± 0.014
[1], DMA, encoder Jacobian0.877 ± 0.0230.925 ± 0.011
[1], IMA, encoder Jacobian0.702 ± 0.0510.932 ± 0.006

On 3DShapes, DSReg performs better than all other methods that read Jacobians. In the world where footprints overlap, DSReg and column support both match the supervised oracle (0.952), but joint diagonalization and IMA contrast both perform worse than the representation with no rotation. Joint diagonalization, IMA contrast and DMA are successful on 3DShapes, likely because the hues of the floor, wall and object change largely separate areas of the image. In our continuous port of 3DShapes, the true Jacobian columns are almost orthogonal, which is close to the structure assumed by these criteria. When footprints overlap and the columns are no longer orthogonal, joint diagonalization and IMA contrast fail; however, DSReg continues to recover the latents.

For [1], we use the losses as they are printed in Equation 1 of [1] and fit them using the same random restarts that were used for the criteria on BaB_a. The two generative baselines remain below each of the Jacobian criteria by a wide margin, even when their hyperparameters are selected for the maximum MCC. Their comparison with DSReg on 3DShapes and quarter dSprites is given below (Table 17, L2828).

Method3DShapesQuarter dSprites
DSReg0.922 ± 0.0050.598 ± 0.079
Jacobian ℓ1\ell_1 VAE [2]0.316 ± 0.0930.442 ± 0.049
Autoencoder with ℓ1\ell_1 sparsity, following [3]0.331 ± 0.0700.342 ± 0.054

In the other worlds of Table 18, DSReg reaches the supervised oracle on both visual suites, and on MPI3D real it is within one standard deviation of the DMA criterion of [1]. Where every footprint is the whole image with orthogonal columns, Structural Diversity fails, and the IMA-type criteria recover the latents while DSReg does not, which is the boundary that Theorem 2 predicts.

[1] Leemann et al., "When are Post-hoc Conceptual Explanations Identifiable?", UAI 2023.

[2] Rhodes and Lee, "Local Disentanglement in Variational Auto-Encoders Using Jacobian L1L_1 Regularization", NeurIPS 2021.

[3] Zheng et al., "Diverse Dictionary Learning", ICLR 2026.

AuthorsOpenReview ↗

We sincerely thank you for your time and thoughtful review (3/4)

Q3. Can the authors establish sufficient conditions under which the population ℓ1\ell_1 objective in Equation 4 has the signed-permutation minimizers identified by Theorem 2? How does the refinement perform systematically where derivative magnitudes make the population ℓ1\ell_1 optimum a mixed rotation?

A3. Thanks so much for the great question. Yes, we have established such conditions (Appendix A.5, L1445). Let D(a)D(a) be the matrix of local slopes at anchor aa, and let E\mathbb{E} average over anchors under any measure, which includes the empirical measure on the sampled anchors. Let wiw_i be the mass of latent ii on the (row, anchor) pairs that it alone drives, let mim_i be its mass on pairs shared with another latent, and let c=E∑r(∥Dr∥1−∥Dr∥2)c=\mathbb{E}\sum_r(\Vert D_r\Vert_1-\Vert D_r\Vert_2).

Proposition 4. If wi>(1+2)miw_i > (1+\sqrt{2})m_i and wi>(3+22)cw_i > (3+2\sqrt{2})c for every ii, then the minimizers of the ℓ1\ell_1 objective over O(d)O(d) are exactly the signed permutations.

Corollary 1. Let D∘D^\circ satisfy this condition, with margins γ=min⁡i((2−1)wi−mi)\gamma=\min_i((\sqrt{2}-1)w_i-m_i) and ξ=(3−22)min⁡iwi−c\xi=(3-2\sqrt{2})\min_i w_i-c, and let D^=D∘+E\widehat D=D^\circ+E with εˉ=E∑r∥Er∥2\bar\varepsilon=\mathbb{E}\sum_r\Vert E_r\Vert_2. If 2dεˉ<ξ2\sqrt{d}\bar\varepsilon < \xi, then every minimizer of Equation 4 on D^\widehat D lies within 2dεˉ/γ2\sqrt{d}\bar\varepsilon/\gamma of a signed permutation, measured as the sum of row-wise ℓ1\ell_1 distances.

Proposition 5. A signed permutation is a strict local minimizer if a first-order inequality holds in every skew direction, and it is not a local minimizer if the inequality is reversed in one direction. This is the rotation analogue of the null space property in ℓ1\ell_1 recovery.

Proposition 4 works at the level of pairs of a row and an anchor, and therefore it also applies when there are latents whose footprints coincide at the global level, provided that each latent retains enough such pairs at the anchors (this is a case outside Theorem 2). In Corollary 1, the only way the error enters is through its average across the anchors. Therefore the error term EE may include ridge error, finite-sample error and representation error, all at the same time. Equation 4 also succeeds beyond the condition. We generated random worlds in which each latent had private rows as well as additional shared rows of growing strength. The ℓ1\ell_1 search reached a signed permutation in all of these worlds, including the worlds that violate the condition. Equation 4 also recovers the diverse and nested regimes from Figure 3b, which contain latents with no private pairs.

For the second question, the refinement repairs this case. We swept the shared entry in gα(z)=(z1+αz2+0.1sin⁡z2,z2)g_\alpha(z)=(z_1+\alpha z_2+0.1\sin z_2,z_2). From α=0.5\alpha=0.5 on, the ℓ1\ell_1 solution is a mixed rotation, and the global count started from it returns the true axes at every α\alpha up to 1.5 and never ends below its starting point. The results (mean MCC, 128 anchors) are as follows.

α\alpha0.10.250.50.751.01.52.0
ℓ1\ell_1, Equation 41.0001.0000.8740.7720.7340.8460.902
Global group ℓ1\ell_11.0001.0000.8780.7780.7290.8430.900
Global count, started from ℓ1\ell_11.0001.0001.0001.0001.0001.0000.961

A larger sweep with N=16N=16 and N=32N=32, using one, two or four shared entries per latent with identical or mixed signs and exactly computed Jacobians, confirms this. The ℓ1\ell_1 solution is mixed in most of these worlds, and when the global count is started from the ℓ1\ell_1 solution, it returns the true axes in almost all of them, with the rest at an MCC of 0.997 or above.

Reviewer_dv6COpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether JEPA world models, which never reconstruct observations, can recover individual world latents rather than a rotated mixture of them. Building on LeJEPA's Gaussian latents up to an orthogonal transformation, the authors introduce structural diversity. The work prove that the rotation minimizing the support of dependency Jacobian recovers each latent up to signed permutation. The resulting post hoc method, DSReg, improves latent recovery and sparse downstream tasks across synthetic and rendered benchmarks while leaving dense performance unchanged.

Critical strengths

The core idea is promising. Separating subspace learning from rotation selection yields a cheap, post hoc method that needs no decoder or retraining. The experiments are thorough, they confirm the predicted failure regimes as well as the successes, and design choices, scale to high dimensions, use held-out splits and multiple seeds.

Critical weaknesses
  1. The paper presents as recovering latents “without reconstruction” and without any link to the observations. But the work looks at the raw observations -- around each anchor point and fits a small linear model that predicts how the pixels change when the representation changes. This is essentially a local, linear decoder, even though it is never trained as one. The method still relies on observation-level information. The authors should avoid implying that no connection to the observations is needed.

  2. The theory requires the hidden factors to be independent Gaussian variables that all change over time at exactly the same rate. The benchmarks satisfy this. Evenv3DShapes and dSprites are rebuilt with Gaussian factors. However, real-world factors are rarely like this -- they are often non-Gaussian, correlated with each other, and may change at different rates. It is therefore unclear how well the method works when the premise is not entirely true.

  3. The paper argues that structural diversity is a weaker requirement than the conditions used in earlier identifiability work. However, the earlier methods solve a harder problem: they must recover the latents from scratch. This work starts from a representation that is already correct up to a rotation, starting from LeJEPA. The comparison should be framed more carefully.

  4. The evidence on realistic data is limited. The pixel-based experiments are designed so that each factor affects a separate, localized part of the image, such as one colored region or one object moving in its own lane. In real images, many important factors, such as camera position, lighting or viewpoint, change almost every pixel, so their footprints may even coincide. Several downstream tests use ideal representations in which the assumptions hold. Experiments on less curated data, or on features from existing pretrained world models, would make the paper's practical claims much more convincing.

Questions for authors

Please see the concerns above.

8 comments
AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback (1/3)

We are very grateful for your insightful and constructive feedback. We are glad that you find the core idea promising, the post hoc method efficient, and the experiments thorough. In light of your suggestions, we have conducted new experiments and added detailed discussions. Please kindly find our point-by-point responses below.


Q1. The method fits a small linear model around each anchor that predicts how the pixels change when the representation changes, which is essentially a local, linear decoder. The authors should avoid implying that no connection to the observations is needed.

A1. Thanks so much for your suggestion. We have highlighted the link to observations in both the introduction and the methodology section (L92-94, L332-334), as well as the abstract. Specifically, we now state: "The estimation of DSReg needs a connection to observations for a ridge regression to approximate the local Jacobians." We have also removed all statements that no connection to the observations is needed. In the updated manuscript, "without reconstruction" means that no global decoder is trained and no reconstruction loss enters the objective.


Q2. The theory requires independent Gaussian factors that change at the same rate, while real-world factors are often non-Gaussian, correlated, and may change at different rates. How well does the method work when the premise is not entirely true?

A2. Thanks a lot for the great question. DSReg has the highest MCC in every world of the table below. Non-Gaussian marginals affect it only slightly, while strong correlation and widely unequal rates lower MCC for every method. In light of your question, we have conducted new experiments that break the premise one axis at a time, through the whole pipeline of a learned encoder followed by DSReg (Appendix C.4, L2369). We cover non-Gaussian marginals (Laplace, uniform, Student t and bimodal) and correlated latents (Toeplitz and block correlation), and we also examine unequal per-latent rates.

Structural Diversity constrains the dependency structure and does not place any requirement on the latent distribution. Therefore, these violations can affect the guarantee of Theorem 2 only by moving the learned representation away from the linear premise h=Qzh=Qz. For this reason, the table also contains the premise residual (the fraction of the variance in the whitened representation that the best orthogonal map of the latents cannot explain), which measures how far the learned representation is from this premise. It is 0 if h=Qzh=Qz holds exactly and increases as the encoder moves away from the premise. In the case of a non-Gaussian world or a world with correlated latents, the premise predicts that the encoder recovers a set of Gaussianized, decorrelated latents, and so the residual is measured against those.

The results (latent MCC at N=8N=8) are summarized below. The correlation rows give the correlation strength, whereas the rate rows give the range of per-latent rates.

AxisWorldPremise residualLeJEPAFastICADSReg
ControlGaussian0.0120.640 ± 0.0310.626 ± 0.0430.994 ± 0.001
MarginalLaplace0.0130.621 ± 0.0380.565 ± 0.0260.969 ± 0.001
MarginalUniform0.0160.635 ± 0.0360.946 ± 0.0170.985 ± 0.000
MarginalStudent t0.0130.606 ± 0.0410.580 ± 0.0300.972 ± 0.002
MarginalBimodal0.0110.618 ± 0.0370.629 ± 0.0320.973 ± 0.000
CorrelationToeplitz, 0.40.0110.622 ± 0.0420.638 ± 0.0330.930 ± 0.002
CorrelationToeplitz, 0.80.0110.528 ± 0.0280.536 ± 0.0350.658 ± 0.006
CorrelationBlock, 0.40.0120.599 ± 0.0380.596 ± 0.0330.910 ± 0.003
CorrelationBlock, 0.80.0240.544 ± 0.0300.533 ± 0.0340.570 ± 0.018
Rates0.93 to 0.950.0120.640 ± 0.0400.628 ± 0.0450.994 ± 0.000
Rates0.85 to 0.950.3060.543 ± 0.0590.619 ± 0.0520.842 ± 0.091
Rates0.5 to 0.950.9280.342 ± 0.0430.379 ± 0.0500.475 ± 0.088

Under the four non-Gaussian marginals, the performance of DSReg is close to that under the Gaussian control. The small gap that remains stems from the Pearson score: LeJEPA represents a non-Gaussian latent through a monotone Gaussianizing transform. Because the rank MCC of DSReg ignores this transform, it matches the control. When the latents are correlated, there is no uncorrelated representation that aligns with zz. Therefore, for every method the correlation lowers MCC. At a correlation strength of 0.4, however, DSReg stays well above LeJEPA. Unequal rates affect MCC through their influence on the premise. In particular, MCC falls only once the premise residual grows. In that regime, the best rotation chosen with labels falls almost as far, and thus most of the drop is due to the representation itself.

AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback (2/3)

Q3. Earlier methods must recover the latents from scratch, while this work starts from a representation that is already correct up to a rotation. The comparison should be framed more carefully.

A3. Thanks so much for your constructive suggestion. We have highlighted it in the updated manuscript accordingly (L300-305):

"It is noteworthy that the considered setups are different: DSReg focuses on the JEPA setup without reconstruction, while previous structural conditions in earlier identifiability work focus on the setup with reconstruction. As a result, the corresponding assumptions differ and the overall conditions and application scenarios are not directly comparable. Therefore, we do not claim our theoretical results are strictly stronger, and just focus on the stand-alone structural condition to provide more intuition on different structural conditions."


Q4. In real images, factors such as camera position, lighting, or viewpoint change almost every pixel, so their footprints may even coincide. Several downstream tests use ideal representations. Experiments on less curated data, or on features from existing pretrained world models, would be more convincing.

A4. Thanks a lot for the great suggestion. DSReg performs better than LeJEPA on photographs of a real robot arm. The same is true for scenes where almost every pixel changes because the camera and light change. DSReg continues to perform better than LeJEPA on corrupted frames, as long as the encoder still carries the factors. We have also extended our theory to the case where footprints coincide as global sets; this extension applies as long as each latent still has a sufficiently large number of pixels that it alone changes at the anchors. In light of your suggestion, we have added these new experiments in the updated appendix (Appendix C.10, L2700).

For global factors in real photographs, the MPI3D real dataset provides photographs of a physical robotic arm with real sensor noise, and a one-step variation in camera height causes essentially every pixel of the image to change. Overall MCC increases from 0.406 for LeJEPA to 0.543 for DSReg, and camera height has the largest gain. The per-factor results are given below.

Factor (MPI3D real)LeJEPADSReg
Camera height0.5420.827
Background color0.5590.829
Object color0.5960.769
Vertical axis0.6090.714
Horizontal axis0.3290.389
Object size0.1820.251
Object shape0.0280.021

The four factors with the lowest MCC are the ones whose pixel changes are the weakest and the most similar to each other, as measured before any training. The full table with all baselines is in the updated appendix (Table 12, L2743).

For coinciding footprints, two latents with the same global footprint can still differ at each anchor, and Equation 4 reads exactly this structure. We now prove a sufficient condition at the level of (row, anchor) pairs that can hold even when global footprints coincide (Proposition 4, Appendix A.5, L1445). When we draw random anchor Jacobians in which every latent touches every observed row, so that all global footprints coincide, Equation 4 returns a signed permutation every time. Nested footprints are also handled: with a global brightness factor whose footprint strictly contains those of four local parts, recovery matches a control in which every factor is local.

AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback (3/3)

For camera and lighting factors, we rendered scenes over textures and two real photographs. The camera had pan, tilt and zoom, while the lamp that lit each scene could move to different positions and have varying levels of gain. In one family of scenes we set directly how many rows pan and tilt share, up to all of them. We run each scene in three modes: using the true latents along with exact Jacobians, the true latents along with estimated Jacobians, and a trained encoder. This allows us to determine whether any loss is due to the criterion, the Jacobian estimate or the encoder. The results (latent MCC) are as follows.

WorldLeJEPADSReg, exact JacobiansDSReg, estimated JacobiansDSReg, trained encoder
Pan and tilt share about a quarter of their rows0.6050.9990.9810.938
Pan and tilt share all their rows0.5940.8910.8750.838
Textured scene, coincident pan and tilt0.5940.8530.7070.663
Real photograph 10.5990.8940.7030.667
Real photograph 20.6080.8440.7700.727
3DShapes0.6451.0000.9760.922

DSReg improves over LeJEPA in every scene. With exact Jacobians, recovery declines gradually as pan and tilt share more rows, without collapsing. On the textured scene and the two photographs, most of the gap between exact Jacobians and the trained encoder already appears with estimated Jacobians, so this part of the gap comes from the Jacobian estimate (Table 13, L2754).

For less curated data, we add fresh sensor noise, moving occluders and scene changes to the frames that both the encoder and the Jacobian estimate see, on 3DShapes and a rendered scene. DSReg stays well above LeJEPA at every level of sensor noise. When we keep the true latents and corrupt only the Jacobian estimate, occluders and scene changes move DSReg only slightly. When the encoder is also trained on frames with occluders or scene changes, DSReg leads LeJEPA until both fall to near zero at the strongest levels, where the encoder itself no longer carries the factors (Table 14, L2773).

We have also marked the downstream tests that use ideal representations as such, next to the learned versions of the control, rollout, and surprise probes (L464-470). Thank you again for your insightful suggestions. We believe these new results address your points, and we would be very grateful if you could take them into account in your evaluation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thanks very much for all the time, effort, and insightful suggestions.

Thanks very much for all the time, effort, and insightful suggestions. We summarize our rebuttals as follows:

Reviewer dv6C: We are glad the reviewer finds the core idea promising, the post hoc method cheap, and the experiments thorough. The reviewer also raises questions concerning the link to observations, how we frame our results against earlier conditions, and what happens when the method is used beyond the premise and on realistic data. We revised the discussion and statements on the first two points as suggested, and we also conducted new experiments on premise violations, real photographs, global camera and light factors, and corrupted frames.

Reviewer TZFx: We thank the reviewer for the feedback. The reviewer finds our identifiability result clean, the support-based argument elegant and the procedure practical. The questions mainly concern the guarantee for the objective we implemented and the relation to three previous works. We established new theoretical results for the ℓ1\ell_1 objective in Equation 4 itself, namely sufficient conditions, a stability bound, and a local test. In addition to this, we added the global ℓ1\ell_1 criterion and the global ℓ0\ell_0 criterion, a theorem-level comparison with [1] and [3], and empirical comparisons with [1], [2] and an autoencoder with ℓ1\ell_1 sparsity following [3], as well as many new experiments. Therefore, we believe the questions have been appropriately addressed with factual evidence.

Reviewer Ey2v: We thank the reviewer for finding the problem important and the theoretical analysis and ablations useful. The questions concern the estimator, the grid renderers, representation error, baselines based on Jacobians and the empirical scope. We provided new results for Equation 4 when the Jacobians are estimated, an analysis of grid renderers together with continuous renderers, a characterization of representation error, comparisons to other methods that use the same Jacobians, as well as new experiments on real photographs, global factors and corrupted frames. Therefore, we believe the questions have been appropriately addressed with factual evidence.

All changes in the updated manuscript are highlighted in blue. Thanks again for all the constructive suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.