- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper identifies a fundamental limitation of Joint-Embedding Predictive Architectures (JEPAs), referred to as the rotation barrier. Due to rotational invariance in the latent representation space, conventional JEPA objectives cannot guarantee coordinate-level factor identifiability. To address this issue, the authors propose AP-JEPA, which replaces the standard dense predictor with an anisotropic (diagonal) predictor. Under whitening, the loss then splits into a rotation-invariant term that fixes the subspace and a cross-talk term that fixes the basis, which gives signed-permutation identifiability when timescales differ. A new regularizer, PairSIGReg, enforces joint Gaussianity of consecutive embeddings and yields global subspace recovery. The closed-form basis step is also applied post hoc to I-JEPA and V-JEPA.
Critical strengths
-
Strong and novel theoretical contribution. The paper provides a rigorous characterization of the rotational symmetry inherent in JEPA objectives and introduces a principled solution through anisotropic predictors. The decomposition of the objective into subspace-selection and basis-selection terms is elegant and provides substantial theoretical insight into representation identifiability. Theoretical guarantees are clearly stated and supported by formal analysis (The decomposition (Thm. 1) is clean, and the 2-D formula (eq. 4) is correct).
-
Excellent alignment between theory and experiments. The empirical results consistently validate the theoretical claims across a diverse set of benchmarks. The distinction between coordinate-level factor recovery and traditional linear-readout performance is clearly demonstrated. The post-hoc application to pretrained JEPA representations also highlights the practical relevance of the proposed method. The paper is honest about regimes of failure: the whitening condition, SIGReg's finite stability window, and tied timescales.
• The ablations separate the predictor's role (basis) from the regularizer's role (subspace).
• The stride sweep (Table 20) and the supervision ledger (D.5.1) are good practice
Critical weaknesses
-
Strong assumptions underlying identifiability guarantees. The main theoretical guarantees rely on assumptions such as independent Gaussian latent factors, distinct temporal correlations, and exact constraint satisfaction. It remains unclear how the proposed framework behaves when these assumptions are significantly violated in realistic environments.
-
Limited validation on large-scale world-model systems. While the benchmark suite is substantial, the training experiments are primarily conducted on relatively controlled environments. Additional evaluation on large transformer-based world models and planning settings would further strengthen the practical impact of the work.
-
Theory and experiments are sometimes in tension.
• Prop. 2 says whitening admits the harmonic whenever ρ₁² ≥ ρₙ. At ρ₁=0.95 that is any ρ₂ ≤ 0.9025, yet Fig. 4(a) and Table 7 show AP+whitening reaching 98% MCC at ρ₂=0.50. The paper itself says whitening fails at (0.97, 0.50). • OU-Spiral (0.95, 0.70) is second-order unstable for SIGReg (ρ₂/ρ₁=0.737 > τ*=0.674), yet AP+SIGReg gets 96.6% MCC. • On MPI3D, ρ₁²≈0.970 < ρₙ=0.985, so the theory predicts SIGReg preserves the span. Yet AP+SIGReg loses a factor (worst-factor R² 7.0%). The explanation that PairSIGReg restores it is post hoc, not derived. • Global statements do not seem to predict where optimization lands. This needs discussion.
-
What the training-time predictor adds is unclear.
• The basis step is AMUSE (B.7). Table 11 and Table 16 show dense-trained features plus the post-hoc rotation almost match trained AP: MCC 95.1 vs 95.7, and shoulder error 10.7–10.9° vs 8.2–9.3°. • The strongest baseline (any method plus post-hoc rotation) is missing from Tables 2–3. • PairSIGReg's edge over SIGReg is within one SD on Cartpole, Isaac3D and the synthetic settings, and nearly so on Reacher (88.8±8.3 vs 94.7±1.9). The only large gain is MPI3D, which is the tied case where AP is neutral.
-
The post-hoc headline (10% → 81%) depends on an engineered pair law. • Table 14 shows the Table 3 pairs give only .63/.41 MCC. The "unlabelled" walks step the known factors at deliberately different rates. • The main text omits this caveat. • "Without information loss" conflicts with Table 4, where K400/SSv2 drop up to 1.7/1.9 points. The text says "at most 0.3 and 1.9", but I-JEPA K400 is −1.7.
-
Baseline fairness and statistics. • External baselines use fixed generic hyperparameters. VICReg at 1.5% R² on MPI3D looks broken rather than weak. • The dense predictor is a per-batch closed-form ridge fit with ~1,024 parameters on 256 samples, with the ridge strength unspecified. It likely overfits the batch, which handicaps the dense control. • The stride (56 on Reacher) is a free design choice, and the results are strongly stride-sensitive. • Five seeds, with no tests and with bolding that ignores overlapping SDs. • Sec. 5.3 says PairSIGReg lifts "no other predictor by over 1 point", but Table 5 shows identity on OU-Spiral going 52.0 → 76.4.
-
"Gap predicts gain" is not clean. Cartpole has a larger gap than Reacher (g=.31 vs .26) but a much smaller gain (+10.5 vs +32.6). Table 20 is non-monotone, and Table 18's ∆ρ (.49) and Fig. 6's g (.26) define the Reacher gap differently.
-
Novelty and related work. • The basis-selection idea is SOBI/AMUSE/tICA, acknowledged only in an appendix while the abstract presents the closed form as new. • Missing: Koopman embeddings with diagonal or block-diagonal dynamics (Lusch et al., 2018), time-contrastive learning, and SlowVAE-type temporal disentanglement.
Questions for authors
Have the authors evaluated AP-JEPA within modern large-scale world-model architectures used for planning and control? If not, what are the primary computational challenges? The authors should position their method relative to recent recursive affine predictors such as LIFS [1,2] and clarify the differences in inductive bias, stability guarantees, and computational requirements. [1]K. Belloulata and A. Belalia, "Multi-Scale Latent Prediction via Learnable Iterated Function Systems," 2026 IEEE International Conference on Image Processing (ICIP), Tampere, Finland, 2026, pp. 1-6, doi: 10.1109/ICIP61757.2026.11630226.
[2]Belloulata, 'Contractive MASO-Generalized Predictors for Stable Latent-Space Learning in JEPA, in Transactions on Machine Learning Research (TMLR), 2026, https://openreview.net/forum?id=k2Z2gPOtlq.
This raises an important question: are the observed improvements specific to anisotropic/diagonal structure, or more generally to imposing structure on the JEPA predictor? A conceptual comparison, and ideally an ablation or experimental comparison under matched parameter/FLOP budgets, would help clarify this point. What is the computational overhead of PairSIGReg relative to SIGReg during large-scale training?
- Why does AP+whitening succeed at (0.95, 0.50) against Prop. 2? Is it initialization, the soft covariance penalty, or encoder capacity? 2. How do dense+post-hoc and BYOL/VICReg+post-hoc compare in Tables 2–3? 3. How were the Table 4 pair laws chosen, and would a practitioner without factor knowledge find them? 4. Can the MPI3D PairSIGReg gain be tied to the ρ=1 scene directions? 5. What are the dense-predictor ridge strength and the baseline tuning budgets?
7 comments
Response to Reviewer MCQE (2/4)
W3 / Q3: Prop. 2 predicts the harmonic under whitening at (0.95, 0.50); why does AP+whitening reach .98?
This matches Prop. 2: at this spectrum the harmonic is the global optimum under whitening, and the factor solution is only a saddle. Training first reaches the factor saddle and leaves it slowly. Table 9 stopped at 12k steps, while the model was still at the saddle (.983). By 40k steps, training escapes to the harmonic optimum, which has strictly lower loss (.16 vs. .43). Thus Q3 is explained by the training trajectory rather than encoder capacity (two architectures behave alike) or a soft penalty (the harmonic satisfies the whitening constraint exactly).
This escape applies only to whitening, which Table 1 already identifies as having the narrowest regime. With SIGReg and PairSIGReg, longer training only stabilizes factor recovery (MCC across training steps):
| Method | 12k steps | 20k steps | 40k steps |
|---|---|---|---|
| AP+whitening | .983 | .970 | .746 |
| AP+SIGReg | .994 | .995 | .995 |
| AP+PairSIGReg | .997 | .997 | .998 |
Their harmonic share stays at .013 or below across the entire Fig. 4(a) sweep. Moreover, escape under whitening is detectable without labels: the replaced coordinate's coefficient drifts from .50 to .84–.91, near ρ₁² = .9025. We have added these trajectories and sweep (Figure 5, Appendix D.1.2) and noted the 12k-step length in Table 9.
W3: OU-Spiral lies outside SIGReg's second-order stability window (Prop. 4); why does AP+SIGReg reach .966?
Second-order instability means a small deformation lowers the loss, not that training moves far from the factor solution. Prop. 4 shows such a direction exists, and its quadratic form measures how steep it is (steepest unstable direction, w = ρ, normalized by ‖b‖²):
| Setting | OU-Spiral | (0.95, 0.60) | (0.97, 0.50) |
|---|---|---|---|
| Quadratic form | .007 | .020 | .068 |
| AP+SIGReg MCC | .966 | .856 | .798 |
On OU-Spiral the curvature is nearly flat (.007), so the deformation stays small: the learned coordinates retain 94% of their variance on the true factors (with ~5% third-order component), yielding a correlation of roughly √.94 ≈ .97, matching the observed .966 MCC (Appendix D.2.3).
W3 / Q6: Why does SIGReg lose an arm factor on MPI3D, and is the PairSIGReg gain tied to the ρ = 1 scene directions?
Yes, the PairSIGReg gain is directly tied to the ρ = 1 scene directions. The calculation in the review considers only the two arm factors. However, MPI3D also contains scene factors (object, camera, background) held fixed within each pair, which therefore have ρ = 1 (Table 22). Including them gives ρ_max² = 1 > ρ_min = .985, so Prop. 2's condition (inherited by SIGReg) fails, whereas Theorem 2 holds for any distinct-timescale spectrum. To verify this, we regenerated MPI3D-real so that the second frame switches to a different scene with probability p (worst-arm-factor R²):
| Scene change within a pair | never (original, Table 12) | p = .25 | p = .5 | always |
|---|---|---|---|---|
| AP+SIGReg | .07 | .91 | .96 | .96 |
| AP+PairSIGReg | .89 | .84 | .73 | .78 |
Once the scene varies within pairs, its variance share under AP+SIGReg drops from 92.5% to 0.5%, SIGReg retains both arm factors, and PairSIGReg is no longer needed (Appendix D.3.3).
W3: Do the global results predict where optimization lands?
Like standard identifiability results (Khemakhem et al., 2020; Zimmermann et al., 2021; von Kügelgen et al., 2021), Theorems 1–2 characterize optima. In addition, the paper provides two practical tools for optimization outcomes: local stability analysis (Prop. 4, Table 1), which flags unstable equilibria such as the whitening saddle, and label-free diagnostics of the trained model via fitted coefficients (Prop. 1), lag eigenvalues (which reveal MPI3D's scene mode at 1.000), and the Prop. 8 perturbation bound (Table 23). We have made this explicit in §5.2.
W7: Does a larger timescale gap predict a larger gain (Cartpole vs. Reacher, Table 20, Δρ vs. g)?
The relationship is not monotone across datasets, and we have revised the text to state only what holds: the gain is positive whenever g > 0 and negative at g = 0 (Table 25). Cartpole and Reacher are not directly comparable because their gains use SIGReg and PairSIGReg respectively, and Cartpole contains a non-factor direction (lag eigenvalue ≈ .87) more predictable than the hinge (.66). We also give distinct names to Δρ (gap between factor blocks, Table 22) and g (gap to the rest of the learned spectrum, Fig. 7).
Response to Reviewer MCQE (3/4)
C. What the training-time predictor and PairSIGReg add (W4; Q2, Q4)
W4 / Q4: How do dense, BYOL, VICReg, and the other methods compare once post-hoc AP is applied to them as well?
Our subspace–basis decomposition naturally yields two routes to basis selection: during training via the diagonal predictor, or after training via the closed-form step (post-hoc AP, Prop. 7). Applying post-hoc AP to every baseline in Tables 3–4 confirms what the theory predicts: post-hoc AP only rotates within the learned span, so it cannot recover factors that a baseline did not retain.
On Reacher, full R² bounds what any rotation can achieve (Prop. 7(iii)):
| Method | Full R² (Table 3) | MCC after post-hoc AP |
|---|---|---|
| Barlow Twins | .458 | .488 |
| LeJEPA | .583 | .491 |
| SimDINO | .604 | .739 |
| VICReg | .648 | .557 |
| BYOL | .675 | .615 |
| dense+SIGReg | .959 | .940 |
| AP+PairSIGReg (native, no post-hoc) | .957 | .947 |
With two labels, the shoulder error of external baselines stays above 77° after post-hoc AP (Table 20), versus 8.9° for native AP+PairSIGReg. Post-hoc AP works only when the span already preserves the factors and timescales differ, which in practice is dense+SIGReg (our own ablation using our regularizer).
Across all five benchmarks (MCC; MPI3D averages its three versions; native AP from Tables 3–4, Reacher + post-hoc from Table 13, other post-hoc columns from the rerun):
| Dataset | AP+PairSIGReg | AP+PairSIGReg + post-hoc | Best other method + post-hoc |
|---|---|---|---|
| Reacher | .947 | .957 | .940 (dense+SIGReg) |
| Cartpole | .699 | .716 | .736 (dense+SIGReg) |
| MPI3D (3 versions) | .404 | .666 | .629 (LeJEPA) |
| Falcor3D | .912 | .756 | .823 (Barlow Twins) |
| Isaac3D | .729 | .818 | .827 (dense+SIGReg) |
When two representations share the same span and have distinct timescales (Cartpole, Isaac3D), post-hoc basis selection reaches a similar basis, as Theorem 1 predicts. Training-time AP and PairSIGReg remain essential in two regimes: (1) Subspace retention on MPI3D, where baselines lose a factor and post-hoc cannot restore it (AP+PairSIGReg reaches .899 full R² against .826 for the best other method, and a rotation cannot raise full R²); and (2) Tied timescales on Falcor3D, where equal lag correlations (.875/.874/.874) give post-hoc no gap to exploit (.756) while training-time AP aligns the axes (.912). We have added this comparison to Appendix D.4.1 (Table 17) and summarized it in §5.1.
W4: What does PairSIGReg add over SIGReg, given that their MCC means are close outside MPI3D?
PairSIGReg targets the factor subspace rather than the basis, and its benefit appears in subspace retention and stability:
| Metric | AP+SIGReg | AP+PairSIGReg |
|---|---|---|
| MPI3D full R² (Table 4) | .408 | .899 |
| MPI3D-real worst-factor R² (Table 12) | .070 | .893 |
| Reacher worst-factor R² (Table 12) | .872 | .928 |
| Reacher MCC SD (Table 3) | .083 | .019 |
Because MPI3D has tied timescales, the predictor cannot select a basis and failure occurs in the subspace; PairSIGReg restores the lost factor across every predictor (Table 12) and also leads in the synthetic sweep of Figure 3(a) where only its guarantee holds.
Q2: Do the gains come from the diagonal structure itself or from any structure on the predictor, at matched parameters?
What drives alignment is the symmetry group of the predictor, not its parameter count (Table 8). Two added controls on Reacher (SIGReg, native MCC) confirm that MCC drops monotonically as the predictor's invariance group grows:
| Predictor | Parameters | Invariant under | MCC |
|---|---|---|---|
| diagonal (AP, Table 3) | n | permutations and sign flips | .888 |
| 2×2 block-diagonal | 2n | rotations within each block | .825 |
| 4×4 block-diagonal | 4n | rotations within each block | .772 |
| dense, closed form (Table 3) | n² | all rotations | .562 |
| dense, trained by SGD | n² | all rotations | .551 |
| shared scalar (Table 7) | 1 | all rotations | .493 |
The 1-parameter scalar is the most constrained yet among the worst, ruling out parameter-count regularization as the cause; both controls are now in Table 8. All closed-form predictors have negligible overhead next to the encoder. In LeWM on Two-Room (one training seed each), goal-image planning reaches 50/50 with the diagonal term alone and 45/50 with the PairSIGReg term alone, against 43/50 for standard LeWM, under identical architecture, parameter count and planning FLOPs.
Response to Reviewer MCQE (1/4)
We thank the reviewer for the detailed review and the positive assessment of our theory.
A. Planning and control with a JEPA world model (W2; Q1, Q2)
Yes. We trained an action-conditioned JEPA world model (LeWM) with our diagonal and PairSIGReg terms and evaluated it on LeWM's official Reacher, Two-Room, and Cube tasks. LeWM (Maes et al., 2026) learns from pixels, predicts the next embedding with an action-conditioned transformer, and plans with CEM in embedding space. Keeping the rest of its training and evaluation recipe unchanged (50 episodes per task):
AP aligns the world model's coordinates across all three tasks while preserving linear decodability: full R² stays at .999–1.000 on Reacher and Two-Room, and .927 vs. .973 on Cube (where cube position error is unchanged at 1.2 vs. 1.3 cm). Because goal-image planning minimizes full-embedding distance (which is rotation-invariant), the gains on Two-Room (where Maes et al., 2026, §4.2 found standard LeWM behind PLDM and DINO-WM) and on Cube (OGBench 3D manipulation) show that the AP terms also yield a stronger world model.
With AP, a few native coordinates selected without labels can steer a single factor. When the goal specifies one factor (a Reacher shoulder angle within 0.1 rad, or a Two-Room x or y coordinate within 16 px), we select the slowest native coordinates by lag autocorrelation (six on Reacher, four on Two-Room) and fit a linear cost using 16 labeled frames (160 start–goal pairs):
The computational overhead is small. Both AP terms are closed-form per batch and store no parameters. At d = 2048 and batch 1024, the loss takes 2.0 ms with SIGReg, 9.8 ms with PairSIGReg, and 18.9 ms for a dense predictor with SIGReg, so the diagonal predictor scales better than a dense one. On Reacher, an epoch takes 3.07 s with AP+SIGReg and 3.76 s with AP+PairSIGReg at identical peak memory. Empirically, the diagonal and PairSIGReg terms transfer directly to nonlinear action-conditioned transformers; extending our linear-predictor identifiability guarantees to action-conditioned predictors is a natural theoretical next step. We have added these results to §5.1 (Table 6) and Appendix D.10 (Tables 26–27). Figure 8 shows official rollouts, including Cube episodes where only LeWM + AP terms guides the 3D robot arm to bring the cube to the goal.
B. Connection between theory and experiments (W1, W3, W7; Q3, Q6)
Selecting the basis does not require Gaussianity, and our real benchmarks violate only Gaussianity. The basis step uses only lag covariance and time reversibility (Thm. 1, Prop. 7(i)), the second-order criterion of AMUSE/SOBI that identifies sources of any distribution (Tong et al., 1991; Belouchrani et al., 1997). Gaussianity enters solely as a target law to exclude nonlinear harmonics, the same role it plays in SIGReg (Balestriero & LeCun, 2025; Maes et al., 2026). Measured on our real benchmarks, factor dynamics are nearly reversible (antisymmetric part ≤ 1.1%) and nearly diagonal (off-diagonal part ≤ 0.6%), deviating only from Gaussianity (excess kurtosis −0.8 to −1.5). Breaking one assumption at a time on the Table 9 setup (ρ = (0.95, 0.50), MCC):
Lighter non-Gaussian noise (t₅, Laplace, uniform) keeps AP above .987, and breaking reversibility has negligible effect. Cross-factor coupling degrades the isotropic baseline sharply, whereas AP+PairSIGReg remains stable because the diagonal predictor and joint regularization still separate the underlying modes.