- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DSReg, a post-hoc method that resolves the rotational ambiguity in linearly identifiable representations by seeking a coordinate system with sparse observation-to-latent Jacobian structure. Under a Gaussian JEPA identifiability premise, distinct dependency footprints, and a functional no-cancellation condition, the authors prove that individual latent factors can be recovered up to sign and permutation. Experiments on synthetic environments, learned visual encoders, and rendered datasets report improved factor-wise recovery and performance on several tasks requiring sparse access to latent variables.
Critical strengths
The paper tackles an important limitation of predictive representations: retaining the complete latent state does not ensure that each coordinate captures a distinct factor of the world. It clearly distinguishes preservation of the overall information from recovery of individual factors. The theoretical analysis formalises the role and limits of dependency structure, while the appendix provides useful ablations studies.
Critical weaknesses
-
The theoretical guarantees do not directly apply to the estimator used in the experiments. Theorem 2 concerns minimising the exact population support count, while Theorem 3 analyses a thresholded population criterion. In practice, the authors optimise an anchor-averaged surrogate based on finite-sample local ridge estimates of the Jacobian. These objectives are not equivalent. Indeed, the paper reports cases in which the objective has a mixed global minimiser even though the exact support criterion identifies the latents. Since exact-count refinement is demonstrated only on small benchmarks, the paper should establish when the practical estimator recovers the theoretically identified solution and report its failure rate and performance relative to the exact criterion.
-
The external-renderer experiments may violate the theorem’s differentiability assumptions. In the 3D Shapes and dSprites experiments, continuous Gaussian latents are mapped onto finite factor grids before rendering. This makes the latent-to-image map piecewise constant, with a Jacobian that is zero almost everywhere. Under the paper’s definition, all dependency footprints would therefore be empty, violating structural diversity. The estimator appears instead to capture finite differences across grid levels. This may provide a useful empirical signal, but it is not the Jacobian support analysed by the theorem. The authors should resolve this mismatch, either by using a continuous generative map or by developing a criterion and analysis for discrete, piecewise-constant observations.
-
The identifiability guarantee relies on a restrictive representation-learning premise. The theorem assumes that the learned representation is exactly an orthogonal transformation of the true Gaussian latent state, under a stationary Gaussian transition and at the population alignment optimum. The learned-encoder experiments approximate this idealised setting, but the robustness analysis considers Jacobian estimation error rather than deviations from linear identifiability. The authors should clarify the limits of the guarantee and characterise how representation error affects recovery before extending the claim to learned encoders.
-
The evaluation lacks comparisons with closely related Jacobian-based methods. The paper discusses Independent Mechanism Analysis (IMA), which can use the same estimated local Jacobians but leaves joint diagonalisation and related baselines to future work. Because DSReg’s main contribution is to use observation-side dependency structure to resolve rotational ambiguity, comparisons with these approaches are important, particularly when their assumptions overlap. The existing comparisons with PCA, Varimax, FastICA, and sparse ICA do not establish whether DSReg outperforms alternative methods that exploit the same observation-side information.
-
The empirical scope is limited relative to the paper’s world-model framing. The learned visual benchmarks use controlled synthetic scenes, and the external-renderer benchmarks use discretised factors. The paper does not evaluate DSReg on real video or physical systems, where noise, occlusion, and changing scene structure could affect local Jacobian estimates. Although the authors acknowledge this limitation, claims about recovering real-world variables should be tempered until validated in more realistic settings.
Questions for authors
Please see the weakness section.
9 comments
We truly appreciate your insightful and constructive suggestions (2/5)
For the experiments, we now separate a criterion failure, where a mixed rotation has a lower objective than the true axes, from an optimizer failure, where the true axes win but are not reached (L346-348, L2189-2191). On the protocol of the submission with exact Jacobians, the results (latent MCC, ) are as follows. The first row is the representation before any rotation is fitted, and the two count rows search, starting from the solution, for the rotation with the fewest nonzero Jacobian entries, counted at each anchor or once over all anchors.
| Criterion | Diverse | Minimal | Nested |
|---|---|---|---|
| No rotation | 0.506 ± 0.028 | 0.506 ± 0.028 | 0.506 ± 0.028 |
| , Equation 4 | 1.000 ± 0.000 | 0.966 ± 0.036 | 1.000 ± 0.000 |
| Count at each anchor, started from | 1.000 ± 0.000 | 1.000 ± 0.000 | 1.000 ± 0.000 |
| Global count, started from | 1.000 ± 0.000 | 1.000 ± 0.000 | 1.000 ± 0.000 |
Started from the solution, the global count, a thresholded form of the support count in Theorem 2, returns the true axes and the true count in every world at and . Where derivative magnitudes make the optimum a mixed rotation, as in at , the same refinement returns the true axes in almost every run and otherwise keeps the solution.
We have also extended this protocol up to . Across the 600 worlds from to , the solution is more than 10 degree away from the true axes in 280 cases, which result from a criterion failure (167 cases) or an optimizer failure (113 cases), mostly at , and its MCC remains 0.92 or above at . Started from the solution after local polishing, the count at each anchor returns the true axes in 594 of the 600 worlds and has an MCC of 0.998 or above in the other six. The results at are as follows.
| Criterion, | Diverse | Minimal | Nested |
|---|---|---|---|
| No rotation | 0.243 ± 0.003 | 0.243 ± 0.003 | 0.243 ± 0.003 |
| , Equation 4 | 0.998 ± 0.001 | 0.947 ± 0.011 | 0.961 ± 0.007 |
| Count at each anchor, started from | 1.000 ± 0.000 | 1.000 ± 0.000 | 1.000 ± 0.000 |
With Jacobians estimated by local ridge regression, the in-sample objective often prefers a mixed rotation, but on held-out anchors the true axes win in almost every such world, so nearly all of these failures come from estimation error on the fitted anchors. With a label-free threshold set from the standard error of the ridge slopes, the refinement raises the diverse family at from 0.876 to 0.999.
As a label-free diagnostic, the distance the refinement moves the solution flags its failures with AUROC 0.99, for exact and estimated Jacobians alike. On the learned benchmarks, where the refinement is kept only if it lowers the count on held-out anchors, it almost never does, so the solution stands, and Equation 4 there is already within 0.005 of the supervised oracle on 3DShapes and both visual suites.
We truly appreciate your insightful and constructive suggestions (3/5)
Q2. The external-renderer experiments may violate the differentiability assumptions. Continuous latents are mapped onto finite factor grids, so the latent-to-image map is piecewise constant, with a Jacobian that is zero almost everywhere. The authors should resolve this mismatch, either with a continuous generative map or with an analysis for discrete observations.
A2. Thanks so much for this insightful point. In light of it, we have taken both routes you suggested, an analysis for grid observations and continuous renderers (Appendix A.6, L1611, and Appendix C.9, L2667). In short, on a grid the local regression estimates slopes with the same footprints as the grid, so our stability results apply to them, and on a continuous renderer of 3DShapes DSReg recovers the factors better than on the published grid.
For the analysis, on a grid the derivative is zero almost everywhere, and the estimator reads finite differences across levels. For a local regression with Gaussian weights on a continuous regressor, Stein's identity shows that the regression estimates the Jacobian of the smoothed map; the smoothed map is the original map convolved with the kernel (that is, ), and this smoothed map is real analytic. The derivative of row in factor at each anchor is a Gaussian-weighted sum of the jumps of coordinate across each level boundary of factor . Hence, the derivative of row in factor is identically zero exactly when row does not change across any level of factor . Therefore, the smoothed map has the same footprints as the grid. An encoder of grid images is constant on each cell of the grid. Given a representation that is quantized along each factor, the population slope is , which is precisely equal to zero outside the footprints. Propositions 4 and 5 and Corollary 1 all hold for any slope field and thus apply directly to both fields. What distinguishes the estimated slopes from those in these two fields (e.g., the neighborhood shape, the ridge penalty, or finite samples) enters Corollary 1 as error.
For the experiments, we replaced the grid lookup with differentiable renderers that keep the structure of both benchmarks. One port keeps a single sprite with shape, scale, orientation, and two positions, and the other keeps a single object in a room with three hues, scale, shape, and orientation. Two split worlds spread the same factors over separate objects as positive controls. Quantizing each factor to levels sweeps from coarse grids through the published grid sizes to the continuous map. The results (DSReg MCC) are as follows.
| Levels per factor | 4 | 8 | 16 | 32 | Published grid | Continuous |
|---|---|---|---|---|---|---|
| 3DShapes port | 0.880 | 0.917 | 0.939 | 0.946 | 0.917 | 0.947 |
| Split sprites | 0.920 | 0.949 | 0.956 | 0.963 | 0.920 | 0.970 |
| Split scene | 0.897 | 0.910 | 0.932 | 0.928 | 0.914 | 0.915 |
On the 3DShapes port, recovery is highest for the continuous map, where DSReg reaches 0.947 against 0.667 for LeJEPA, and the published grid scores below it, so the grid does not inflate the result. DSReg improves over LeJEPA at every level on all four worlds, including the dSprites port. Scored against the true factors, rotations fitted to local ridge Jacobians also stay within 0.03 of rotations fitted to autograd Jacobians of the continuous renderers on all four worlds. In the manuscript, the external renderers are now presented together with this analysis, as empirical tests beyond the exact conditions of Theorem 2.
We truly appreciate your insightful and constructive suggestions (1/5)
We truly appreciate your insightful and constructive suggestions. We are glad that you find the problem important, the distinction between preserving information and recovering individual factors clear, and the theoretical analysis and ablations useful. In light of your suggestions, we have added new theoretical results, new experiments, and detailed discussions. Please find our detailed responses as follows.
Q1. The guarantees do not directly apply to the estimator. Theorems 2 and 3 concern the exact support count and a thresholded criterion, while in practice the authors optimise an anchor-averaged ℓ1 surrogate on finite-sample ridge estimates, and the ℓ1 objective can have a mixed global minimiser. The paper should establish when the practical estimator recovers the identified solution and report its failure rate and performance relative to the exact criterion.
A1. Thanks a lot for raising this. Accordingly, we have proved new results for Equation 4 itself, including under finite-sample ridge estimates (Appendix A.5, L1445), and we now report how the estimator fails relative to the exact criterion. In short, under a sufficient condition the minimizers of Equation 4 are exactly the signed permutations and stay close to them under small Jacobian error, and in experiments with exact Jacobians a support count started from the ℓ1 solution returns the true axes in almost every world.
In the original version, we mainly reported the anchor-averaged ℓ1 objective since ℓ1 is the standard relaxation of ℓ0, and the local form is mainly for scalability to large models. This is also the case in previous sparsity-based identifiability results, which state the theorem through the support and estimate with ℓ1-type surrogates. For instance, [4] uses an ℓ1 norm on the Jacobian as an approximation of ℓ0 during estimation, and [5] states its theorem through ℓ0 and estimates with continuous surrogates such as ℓ1 and SCAD.
For the theory, let D(a) be the matrix of local slopes at anchor a, under any measure on anchors, so the statements cover the empirical measure on the m anchors. Let wi be the mass of latent i on the (row, anchor) pairs that it alone drives, let mi be its mass on pairs shared with another latent, and let c=E∑r(∥Dr∥1−∥Dr∥2).
Proposition 4. If wi>(1+2)mi and wi>(3+22)c for every i, then the minimizers of the ℓ1 objective over O(d) are exactly the signed permutations.
Corollary 1. Let D∘ satisfy this condition, with margins γ=mini((2−1)wi−mi) and ξ=(3−22)miniwi−c, and let D=D∘+E with εˉ=E∑r∥Er∥2. If 2dεˉ<ξ, then every minimizer of Equation 4 on D lies within 2dεˉ/γ of a signed permutation, measured as the sum of row-wise ℓ1 distances.
Proposition 5. A signed permutation is a strict local minimizer if a first-order inequality holds in every skew direction, and it is not a local minimizer if the inequality is reversed in one direction.
The error E enters only via its mean across anchors. Therefore, it can contain ridge error, finite-sample error and representation error simultaneously. The condition is both sufficient and far from being necessary. In the random worlds where each latent has its own private rows and shared rows are added with increasing strength, the ℓ1 search found a signed permutation every time, even for the worlds that lie outside this condition.
[4] Zheng et al., "On the Identifiability of Nonlinear ICA: Sparsity and Beyond", NeurIPS 2022.
[5] Ng et al., "On the Identifiability of Sparse ICA without Assuming Non-Gaussianity", NeurIPS 2023.