acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Rotation Barrier in JEPA World Models with Anisotropic Predictors

Abstract

Joint-Embedding Predictive Architectures (JEPAs) learn world models by predicting in representation space, but their predictor can absorb any rotation of that space without changing the loss, leaving physical factors mixed across coordinates and hard to monitor or control individually. Recovering one factor per coordinate requires the right subspace, which contains the factors, and the right basis, which gives each factor its own coordinate. We break this rotation barrier with a minimal change, replacing the dense predictor by an anisotropic predictor (**AP-JEPA**), and prove that its loss splits exactly into two terms: a rotation-invariant ***dense-predictor term***, which selects the subspace, and a ***cross-talk term***, which selects the basis and aligns coordinates with factors that evolve on distinct timescales. However, the encoder can produce features that are more predictable than genuine factors, such as the square of a slow factor, so the selected subspace may miss some factors. We characterize when two existing regularizers, whitening and SIGReg, rule out such features, and propose a new regularizer, PairSIGReg, which constrains consecutive representations jointly and recovers the factor subspace for every spectrum of distinct timescales. Across six dynamical and image benchmarks, AP-JEPA with PairSIGReg raises the average mean correlation coefficient (MCC) from for the strongest baseline to , while also achieving the highest average linear readout (). Beyond training from scratch, AP-JEPA also applies to existing JEPAs trained with dense predictors: since the cross-talk term depends only on the basis, minimizing it alone yields a closed-form change of coordinates that raises the single-coordinate of released I-JEPA and V-JEPA models from about to without information loss.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, World Models, Identification, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MCQEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a fundamental limitation of Joint-Embedding Predictive Architectures (JEPAs), referred to as the rotation barrier. Due to rotational invariance in the latent representation space, conventional JEPA objectives cannot guarantee coordinate-level factor identifiability. To address this issue, the authors propose AP-JEPA, which replaces the standard dense predictor with an anisotropic (diagonal) predictor. Under whitening, the loss then splits into a rotation-invariant term that fixes the subspace and a cross-talk term that fixes the basis, which gives signed-permutation identifiability when timescales differ. A new regularizer, PairSIGReg, enforces joint Gaussianity of consecutive embeddings and yields global subspace recovery. The closed-form basis step is also applied post hoc to I-JEPA and V-JEPA.

Critical strengths
  1. Strong and novel theoretical contribution. The paper provides a rigorous characterization of the rotational symmetry inherent in JEPA objectives and introduces a principled solution through anisotropic predictors. The decomposition of the objective into subspace-selection and basis-selection terms is elegant and provides substantial theoretical insight into representation identifiability. Theoretical guarantees are clearly stated and supported by formal analysis (The decomposition (Thm. 1) is clean, and the 2-D formula (eq. 4) is correct).

  2. Excellent alignment between theory and experiments. The empirical results consistently validate the theoretical claims across a diverse set of benchmarks. The distinction between coordinate-level factor recovery and traditional linear-readout performance is clearly demonstrated. The post-hoc application to pretrained JEPA representations also highlights the practical relevance of the proposed method. The paper is honest about regimes of failure: the whitening condition, SIGReg's finite stability window, and tied timescales.

    • The ablations separate the predictor's role (basis) from the regularizer's role (subspace).

    • The stride sweep (Table 20) and the supervision ledger (D.5.1) are good practice

Critical weaknesses
  1. Strong assumptions underlying identifiability guarantees. The main theoretical guarantees rely on assumptions such as independent Gaussian latent factors, distinct temporal correlations, and exact constraint satisfaction. It remains unclear how the proposed framework behaves when these assumptions are significantly violated in realistic environments.

  2. Limited validation on large-scale world-model systems. While the benchmark suite is substantial, the training experiments are primarily conducted on relatively controlled environments. Additional evaluation on large transformer-based world models and planning settings would further strengthen the practical impact of the work.

  3. Theory and experiments are sometimes in tension.

    • Prop. 2 says whitening admits the harmonic whenever ρ₁² ≥ ρₙ. At ρ₁=0.95 that is any ρ₂ ≤ 0.9025, yet Fig. 4(a) and Table 7 show AP+whitening reaching 98% MCC at ρ₂=0.50. The paper itself says whitening fails at (0.97, 0.50). • OU-Spiral (0.95, 0.70) is second-order unstable for SIGReg (ρ₂/ρ₁=0.737 > τ*=0.674), yet AP+SIGReg gets 96.6% MCC. • On MPI3D, ρ₁²≈0.970 < ρₙ=0.985, so the theory predicts SIGReg preserves the span. Yet AP+SIGReg loses a factor (worst-factor R² 7.0%). The explanation that PairSIGReg restores it is post hoc, not derived. • Global statements do not seem to predict where optimization lands. This needs discussion.

  4. What the training-time predictor adds is unclear.

    • The basis step is AMUSE (B.7). Table 11 and Table 16 show dense-trained features plus the post-hoc rotation almost match trained AP: MCC 95.1 vs 95.7, and shoulder error 10.7–10.9° vs 8.2–9.3°. • The strongest baseline (any method plus post-hoc rotation) is missing from Tables 2–3. • PairSIGReg's edge over SIGReg is within one SD on Cartpole, Isaac3D and the synthetic settings, and nearly so on Reacher (88.8±8.3 vs 94.7±1.9). The only large gain is MPI3D, which is the tied case where AP is neutral.

  5. The post-hoc headline (10% → 81%) depends on an engineered pair law. • Table 14 shows the Table 3 pairs give only .63/.41 MCC. The "unlabelled" walks step the known factors at deliberately different rates. • The main text omits this caveat. • "Without information loss" conflicts with Table 4, where K400/SSv2 drop up to 1.7/1.9 points. The text says "at most 0.3 and 1.9", but I-JEPA K400 is −1.7.

  6. Baseline fairness and statistics. • External baselines use fixed generic hyperparameters. VICReg at 1.5% R² on MPI3D looks broken rather than weak. • The dense predictor is a per-batch closed-form ridge fit with ~1,024 parameters on 256 samples, with the ridge strength unspecified. It likely overfits the batch, which handicaps the dense control. • The stride (56 on Reacher) is a free design choice, and the results are strongly stride-sensitive. • Five seeds, with no tests and with bolding that ignores overlapping SDs. • Sec. 5.3 says PairSIGReg lifts "no other predictor by over 1 point", but Table 5 shows identity on OU-Spiral going 52.0 → 76.4.

  7. "Gap predicts gain" is not clean. Cartpole has a larger gap than Reacher (g=.31 vs .26) but a much smaller gain (+10.5 vs +32.6). Table 20 is non-monotone, and Table 18's ∆ρ (.49) and Fig. 6's g (.26) define the Reacher gap differently.

  8. Novelty and related work. • The basis-selection idea is SOBI/AMUSE/tICA, acknowledged only in an appendix while the abstract presents the closed form as new. • Missing: Koopman embeddings with diagonal or block-diagonal dynamics (Lusch et al., 2018), time-contrastive learning, and SlowVAE-type temporal disentanglement.

Questions for authors

Have the authors evaluated AP-JEPA within modern large-scale world-model architectures used for planning and control? If not, what are the primary computational challenges? The authors should position their method relative to recent recursive affine predictors such as LIFS [1,2] and clarify the differences in inductive bias, stability guarantees, and computational requirements. [1]K. Belloulata and A. Belalia, "Multi-Scale Latent Prediction via Learnable Iterated Function Systems," 2026 IEEE International Conference on Image Processing (ICIP), Tampere, Finland, 2026, pp. 1-6, doi: 10.1109/ICIP61757.2026.11630226.

[2]Belloulata, 'Contractive MASO-Generalized Predictors for Stable Latent-Space Learning in JEPA, in Transactions on Machine Learning Research (TMLR), 2026, https://openreview.net/forum?id=k2Z2gPOtlq.

This raises an important question: are the observed improvements specific to anisotropic/diagonal structure, or more generally to imposing structure on the JEPA predictor? A conceptual comparison, and ideally an ablation or experimental comparison under matched parameter/FLOP budgets, would help clarify this point. What is the computational overhead of PairSIGReg relative to SIGReg during large-scale training?

  1. Why does AP+whitening succeed at (0.95, 0.50) against Prop. 2? Is it initialization, the soft covariance penalty, or encoder capacity? 2. How do dense+post-hoc and BYOL/VICReg+post-hoc compare in Tables 2–3? 3. How were the Table 4 pair laws chosen, and would a practitioner without factor knowledge find them? 4. Can the MPI3D PairSIGReg gain be tied to the ρ=1 scene directions? 5. What are the dense-predictor ridge strength and the baseline tuning budgets?
7 comments
AuthorsOpenReview ↗

Response to Reviewer MCQE (1/4)

We thank the reviewer for the detailed review and the positive assessment of our theory.

A. Planning and control with a JEPA world model (W2; Q1, Q2)

W2 / Q1 / Q2: Has AP-JEPA been evaluated in world models used for planning and control, what are the computational challenges, and what does PairSIGReg cost relative to SIGReg?

Yes. We trained an action-conditioned JEPA world model (LeWM) with our diagonal and PairSIGReg terms and evaluated it on LeWM's official Reacher, Two-Room, and Cube tasks. LeWM (Maes et al., 2026) learns from pixels, predicts the next embedding with an action-conditioned transformer, and plans with CEM in embedding space. Keeping the rest of its training and evaluation recipe unchanged (50 episodes per task):

Standard LeWMLeWM + AP terms (ours)
Goal-image planning success
Reacher82%86%
Two-Room85%100%
Cube (3D robot arm)72.7%78.0%
Coordinate alignment (native MCC)
Reacher.443.772
Two-Room.399.795
Cube (3D robot arm).352.448

AP aligns the world model's coordinates across all three tasks while preserving linear decodability: full R² stays at .999–1.000 on Reacher and Two-Room, and .927 vs. .973 on Cube (where cube position error is unchanged at 1.2 vs. 1.3 cm). Because goal-image planning minimizes full-embedding distance (which is rotation-invariant), the gains on Two-Room (where Maes et al., 2026, §4.2 found standard LeWM behind PLDM and DINO-WM) and on Cube (OGBench 3D manipulation) show that the AP terms also yield a stronger world model.

With AP, a few native coordinates selected without labels can steer a single factor. When the goal specifies one factor (a Reacher shoulder angle within 0.1 rad, or a Two-Room x or y coordinate within 16 px), we select the slowest native coordinates by lag autocorrelation (six on Reacher, four on Two-Room) and fit a linear cost using 16 labeled frames (160 start–goal pairs):

Single-factor control taskStandard LeWMLeWM + AP terms (ours)
Reacher (shoulder angle)9.0%31.5%
Two-Room (x coordinate)9.9%35.3%
Two-Room (y coordinate)22.3%34.3%

The computational overhead is small. Both AP terms are closed-form per batch and store no parameters. At d = 2048 and batch 1024, the loss takes 2.0 ms with SIGReg, 9.8 ms with PairSIGReg, and 18.9 ms for a dense predictor with SIGReg, so the diagonal predictor scales better than a dense one. On Reacher, an epoch takes 3.07 s with AP+SIGReg and 3.76 s with AP+PairSIGReg at identical peak memory. Empirically, the diagonal and PairSIGReg terms transfer directly to nonlinear action-conditioned transformers; extending our linear-predictor identifiability guarantees to action-conditioned predictors is a natural theoretical next step. We have added these results to §5.1 (Table 6) and Appendix D.10 (Tables 26–27). Figure 8 shows official rollouts, including Cube episodes where only LeWM + AP terms guides the 3D robot arm to bring the cube to the goal.

B. Connection between theory and experiments (W1, W3, W7; Q3, Q6)

W1: How does AP behave when Gaussianity, diagonal dynamics, or reversibility do not hold?

Selecting the basis does not require Gaussianity, and our real benchmarks violate only Gaussianity. The basis step uses only lag covariance and time reversibility (Thm. 1, Prop. 7(i)), the second-order criterion of AMUSE/SOBI that identifies sources of any distribution (Tong et al., 1991; Belouchrani et al., 1997). Gaussianity enters solely as a target law to exclude nonlinear harmonics, the same role it plays in SIGReg (Balestriero & LeCun, 2025; Maes et al., 2026). Measured on our real benchmarks, factor dynamics are nearly reversible (antisymmetric part ≤ 1.1%) and nearly diagonal (off-diagonal part ≤ 0.6%), deviating only from Gaussianity (excess kurtosis −0.8 to −1.5). Breaking one assumption at a time on the Table 9 setup (ρ = (0.95, 0.50), MCC):

Assumption violated in the dataLeJEPAAP+SIGRegAP+PairSIGReg
none (all assumptions hold).937.995.997
Gaussianity: heavy-tailed t₃ noise.907.938.934
diagonal dynamics: coupling c = 0.1.929.989.992
diagonal dynamics: coupling c = 0.3.798.957.982
reversibility: rotation c = 0.1.942.995.997
reversibility: rotation c = 0.3.956.990.993

Lighter non-Gaussian noise (t₅, Laplace, uniform) keeps AP above .987, and breaking reversibility has negligible effect. Cross-factor coupling degrades the isotropic baseline sharply, whereas AP+PairSIGReg remains stable because the diagonal predictor and joint regularization still separate the underlying modes.

AuthorsOpenReview ↗

Response to Reviewer MCQE (2/4)

W3 / Q3: Prop. 2 predicts the harmonic under whitening at (0.95, 0.50); why does AP+whitening reach .98?

This matches Prop. 2: at this spectrum the harmonic is the global optimum under whitening, and the factor solution is only a saddle. Training first reaches the factor saddle and leaves it slowly. Table 9 stopped at 12k steps, while the model was still at the saddle (.983). By 40k steps, training escapes to the harmonic optimum, which has strictly lower loss (.16 vs. .43). Thus Q3 is explained by the training trajectory rather than encoder capacity (two architectures behave alike) or a soft penalty (the harmonic satisfies the whitening constraint exactly).

This escape applies only to whitening, which Table 1 already identifies as having the narrowest regime. With SIGReg and PairSIGReg, longer training only stabilizes factor recovery (MCC across training steps):

Method12k steps20k steps40k steps
AP+whitening.983.970.746
AP+SIGReg.994.995.995
AP+PairSIGReg.997.997.998

Their harmonic share stays at .013 or below across the entire Fig. 4(a) sweep. Moreover, escape under whitening is detectable without labels: the replaced coordinate's coefficient drifts from .50 to .84–.91, near ρ₁² = .9025. We have added these trajectories and sweep (Figure 5, Appendix D.1.2) and noted the 12k-step length in Table 9.

W3: OU-Spiral lies outside SIGReg's second-order stability window (Prop. 4); why does AP+SIGReg reach .966?

Second-order instability means a small deformation lowers the loss, not that training moves far from the factor solution. Prop. 4 shows such a direction exists, and its quadratic form measures how steep it is (steepest unstable direction, w = ρ, normalized by ‖b‖²):

SettingOU-Spiral(0.95, 0.60)(0.97, 0.50)
Quadratic form.007.020.068
AP+SIGReg MCC.966.856.798

On OU-Spiral the curvature is nearly flat (.007), so the deformation stays small: the learned coordinates retain 94% of their variance on the true factors (with ~5% third-order component), yielding a correlation of roughly √.94 ≈ .97, matching the observed .966 MCC (Appendix D.2.3).

W3 / Q6: Why does SIGReg lose an arm factor on MPI3D, and is the PairSIGReg gain tied to the ρ = 1 scene directions?

Yes, the PairSIGReg gain is directly tied to the ρ = 1 scene directions. The calculation in the review considers only the two arm factors. However, MPI3D also contains scene factors (object, camera, background) held fixed within each pair, which therefore have ρ = 1 (Table 22). Including them gives ρ_max² = 1 > ρ_min = .985, so Prop. 2's condition (inherited by SIGReg) fails, whereas Theorem 2 holds for any distinct-timescale spectrum. To verify this, we regenerated MPI3D-real so that the second frame switches to a different scene with probability p (worst-arm-factor R²):

Scene change within a pairnever (original, Table 12)p = .25p = .5always
AP+SIGReg.07.91.96.96
AP+PairSIGReg.89.84.73.78

Once the scene varies within pairs, its variance share under AP+SIGReg drops from 92.5% to 0.5%, SIGReg retains both arm factors, and PairSIGReg is no longer needed (Appendix D.3.3).

W3: Do the global results predict where optimization lands?

Like standard identifiability results (Khemakhem et al., 2020; Zimmermann et al., 2021; von Kügelgen et al., 2021), Theorems 1–2 characterize optima. In addition, the paper provides two practical tools for optimization outcomes: local stability analysis (Prop. 4, Table 1), which flags unstable equilibria such as the whitening saddle, and label-free diagnostics of the trained model via fitted coefficients (Prop. 1), lag eigenvalues (which reveal MPI3D's scene mode at 1.000), and the Prop. 8 perturbation bound (Table 23). We have made this explicit in §5.2.

W7: Does a larger timescale gap predict a larger gain (Cartpole vs. Reacher, Table 20, Δρ vs. g)?

The relationship is not monotone across datasets, and we have revised the text to state only what holds: the gain is positive whenever g > 0 and negative at g = 0 (Table 25). Cartpole and Reacher are not directly comparable because their gains use SIGReg and PairSIGReg respectively, and Cartpole contains a non-factor direction (lag eigenvalue ≈ .87) more predictable than the hinge (.66). We also give distinct names to Δρ (gap between factor blocks, Table 22) and g (gap to the rest of the learned spectrum, Fig. 7).

AuthorsOpenReview ↗

Response to Reviewer MCQE (3/4)

C. What the training-time predictor and PairSIGReg add (W4; Q2, Q4)

W4 / Q4: How do dense, BYOL, VICReg, and the other methods compare once post-hoc AP is applied to them as well?

Our subspace–basis decomposition naturally yields two routes to basis selection: during training via the diagonal predictor, or after training via the closed-form step (post-hoc AP, Prop. 7). Applying post-hoc AP to every baseline in Tables 3–4 confirms what the theory predicts: post-hoc AP only rotates within the learned span, so it cannot recover factors that a baseline did not retain.

On Reacher, full R² bounds what any rotation can achieve (Prop. 7(iii)):

MethodFull R² (Table 3)MCC after post-hoc AP
Barlow Twins.458.488
LeJEPA.583.491
SimDINO.604.739
VICReg.648.557
BYOL.675.615
dense+SIGReg.959.940
AP+PairSIGReg (native, no post-hoc).957.947

With two labels, the shoulder error of external baselines stays above 77° after post-hoc AP (Table 20), versus 8.9° for native AP+PairSIGReg. Post-hoc AP works only when the span already preserves the factors and timescales differ, which in practice is dense+SIGReg (our own ablation using our regularizer).

Across all five benchmarks (MCC; MPI3D averages its three versions; native AP from Tables 3–4, Reacher + post-hoc from Table 13, other post-hoc columns from the rerun):

DatasetAP+PairSIGRegAP+PairSIGReg + post-hocBest other method + post-hoc
Reacher.947.957.940 (dense+SIGReg)
Cartpole.699.716.736 (dense+SIGReg)
MPI3D (3 versions).404.666.629 (LeJEPA)
Falcor3D.912.756.823 (Barlow Twins)
Isaac3D.729.818.827 (dense+SIGReg)

When two representations share the same span and have distinct timescales (Cartpole, Isaac3D), post-hoc basis selection reaches a similar basis, as Theorem 1 predicts. Training-time AP and PairSIGReg remain essential in two regimes: (1) Subspace retention on MPI3D, where baselines lose a factor and post-hoc cannot restore it (AP+PairSIGReg reaches .899 full R² against .826 for the best other method, and a rotation cannot raise full R²); and (2) Tied timescales on Falcor3D, where equal lag correlations (.875/.874/.874) give post-hoc no gap to exploit (.756) while training-time AP aligns the axes (.912). We have added this comparison to Appendix D.4.1 (Table 17) and summarized it in §5.1.

W4: What does PairSIGReg add over SIGReg, given that their MCC means are close outside MPI3D?

PairSIGReg targets the factor subspace rather than the basis, and its benefit appears in subspace retention and stability:

MetricAP+SIGRegAP+PairSIGReg
MPI3D full R² (Table 4).408.899
MPI3D-real worst-factor R² (Table 12).070.893
Reacher worst-factor R² (Table 12).872.928
Reacher MCC SD (Table 3).083.019

Because MPI3D has tied timescales, the predictor cannot select a basis and failure occurs in the subspace; PairSIGReg restores the lost factor across every predictor (Table 12) and also leads in the synthetic sweep of Figure 3(a) where only its guarantee holds.

Q2: Do the gains come from the diagonal structure itself or from any structure on the predictor, at matched parameters?

What drives alignment is the symmetry group of the predictor, not its parameter count (Table 8). Two added controls on Reacher (SIGReg, native MCC) confirm that MCC drops monotonically as the predictor's invariance group grows:

PredictorParametersInvariant underMCC
diagonal (AP, Table 3)npermutations and sign flips.888
2×2 block-diagonal2nrotations within each block.825
4×4 block-diagonal4nrotations within each block.772
dense, closed form (Table 3)n²all rotations.562
dense, trained by SGDn²all rotations.551
shared scalar (Table 7)1all rotations.493

The 1-parameter scalar is the most constrained yet among the worst, ruling out parameter-count regularization as the cause; both controls are now in Table 8. All closed-form predictors have negligible overhead next to the encoder. In LeWM on Two-Room (one training seed each), goal-image planning reaches 50/50 with the diagonal term alone and 45/50 with the PairSIGReg term alone, against 43/50 for standard LeWM, under identical architecture, parameter count and planning FLOPs.

Reviewer_1KAqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce an anisotropic predictor, in place of a dense predictor, to break the rotation barrier common in JEPA world models. By rotation barrier, they mean that the prediction objective is invariant to rotations of the representation space, preventing the identification of individual factors in separate coordinates. The paper decomposes the prediction loss into a rotation-invariant term, which selects the subspace, and a cross-talk term, which selects the basis. It also introduces PairSIGReg to address limitations of whitening and SIGReg in recovering the correct factor subspace. The method is evaluated on six dynamical and image benchmarks, with additional experiments on I-JEPA and V-JEPA. in general, the results show improvements in factor recovery.

Critical strengths
  1. This is an important problem in a relevant field. The paper investigates the identifiability of representations learned by JEPA world models. The distinction between subspace and basis the authors provide is interesting and provides some useful perspective on the problem.
  2. I find the decomposition of the prediction loss is elegant. The characterisation of when whitening and SIGReg recover the factors, together with the proposed PairSIGReg regulariser, provides a more complete understanding of the conditions needed for identifiability.
Critical weaknesses
  1. It is not well motivated why separating rotation from the predictor useful. The paper claims benefits for monitoring and control, but does not demonstrate them. Table 4 shows substantial improvements in factor recovery, without explicitely motivating or demonstrating when such features could help. Related to this, I wonder why the authors consider only raotations, and not the more general case of linear identifiability, like the community is prioritising [1,2].
  2. The theoretical guarantees rely on strict assumptions that I am not sure matter or are verified experimentally (e.g., independent Gaussian factors, reversible dynamics, distinct timescales, and exact regularisation constraints). In particular, it is unclear whether Figure 3(a) satisfies the assumptions of Theorem 2. It is not expected that the assumptions match the experiments, but a path to how those could be made to agree would be useful, see [1] Proposition 2 for instance.
  3. L933: the authors mention "our closed-form map for frozen JEPAs coincides with the AMUSE rotation", which weakens the contribution; specifically, the originally of this method is not well established.

References:

[1] Nzoyem et al. "Neural Context Flows for Meta-Learning of Dynamical Systems", ICLR 2025.
[2] Blanke and Lelarge, "Interpretable Meta-Learning of Physical Systems", ICLR 2024.

Questions for authors

See the weaknesses above for more context on the questions below:

  1. Beyond Table 4, can you demonstrate that improved factor recovery benefits a concrete downstream task, such as planning or control?
  2. To what extent do the experiments satisfy the assumptions of Theorem 2? In particular, are the learned representations jointly Gaussian?
  3. Can you clarify how/if the method uses natural temporal transitions (i.e. doesn't always construct pairs from ground-truth factors)? Also, how does the proposed post-hoc approach differ from AMUSE?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 1KAq (1/2)

We thank the reviewer for the thoughtful review and for finding the decomposition elegant and the characterization of the regularizers useful. Below, we address the questions on downstream utility, linear identifiability, theoretical assumptions, and AMUSE.

W1 / Q1: Beyond Table 4, does factor recovery help a downstream task such as planning or control?

Isolating factors into individual coordinates matters whenever a task monitors or steers a single physical factor with minimal supervision. When coordinates are mixed by a rotation, reading out one joint angle on Reacher with a full-dimensional probe needs ~120 labeled frames to match what AP achieves with just two labels on a single coordinate (8.9° for AP+PairSIGReg with two labels, against more than 75° for every rotated baseline; Figure 3(c), Table 20).

To test closed-loop planning and control, we added our diagonal and PairSIGReg terms to LeWM (Maes et al., 2026). LeWM learns from pixels, predicts the next embedding with an action-conditioned transformer, and plans with CEM in embedding space. Keeping the rest of its training and evaluation recipe unchanged (50 episodes per task):

Standard LeWMLeWM + AP terms (ours)
Goal-image planning success
Reacher82%86%
Two-Room85%100%
Cube (3D robot arm)72.7%78.0%
Coordinate alignment (native MCC)
Reacher.443.772
Two-Room.399.795
Cube (3D robot arm).352.448

AP aligns the world model's coordinates on all three environments while preserving linear decodability: full R² stays at .999–1.000 on Reacher and Two-Room, and .927 vs. .973 on Cube (where cube position error is unchanged at 1.2 vs. 1.3 cm). Because goal-image planning uses full-embedding distance (which is rotation-invariant), the gains on Two-Room (where Maes et al., 2026 found standard LeWM behind PLDM and DINO-WM) and on Cube (OGBench 3D manipulation) show that the AP terms also produce a stronger world model.

The direct benefit of coordinate alignment appears when steering a single physical factor with few labels. When the goal is one factor (a Reacher shoulder angle within 0.1 rad, or a Two-Room x or y position within 16 px), we select the slowest native coordinates without labels via lag autocorrelation and fit a linear cost using 16 labeled frames (160 start–goal pairs):

Single-factor control taskStandard LeWMLeWM + AP terms (ours)
Reacher (shoulder angle)9.0%31.5%
Two-Room (x coordinate)9.9%35.3%
Two-Room (y coordinate)22.3%34.3%

We have added these experiments to §5.1 (Table 6) and Appendix D.10 (Tables 26–27). Figure 8 shows official rollouts, including Cube episodes where only LeWM + AP terms guides the 3D robot arm to bring the cube to the goal.

W1: Why target coordinates, i.e. rotations, rather than general linear identifiability [1, 2]?

Coordinate identifiability is the strongest of the three linear levels and implies the other two:

signed permutations  ⊂  rotations O(n)  ⊂  invertible linear maps GL(n)
   coordinate        ⇒    orthogonal    ⇒          linear
   (this work)          (whitening; JEPA theory)    ([1, 2])

In multi-system meta-learning [1, 2], GL(n) identifiability suffices because each new context adapts a linear readout. In a single regularized world model, however, whitening already reduces GL(n) ambiguity to an orthogonal rotation O(n) (Prop. 7(ii); Klindt et al., 2026; Zhang et al., 2026). Removing that remaining rotation is what enables few-label readout and single-factor control on individual coordinates. We have added this hierarchy to §1 and discussed [1, 2] in §2 as complementary.

W2 / Q2: How far do the experiments satisfy the assumptions of Theorem 2, and are the learned representations jointly Gaussian?

Real benchmarks nearly satisfy the dynamical assumptions that select the basis and deviate only from Gaussianity, which the theory uses solely to rule out harmonics. Following [1, Prop. 2], we check each assumption on real data, break assumptions one at a time in ablations, and verify the theory's bounds on non-Gaussian benchmarks.

In linear maps, Theorem 1's basis-selection term depends only on lag covariance and time reversibility, not on Gaussianity. Across our real benchmarks, factor transitions are nearly reversible (antisymmetric part ≤ 1.1%) and nearly diagonal (off-diagonal part ≤ 0.6%), with distinct timescales on native-dynamics tasks like Reacher and Cartpole (on the constructed MPI3D and Falcor3D pairs the scored factors share one timescale, where Prop. 1 guarantees only a block). Only Gaussianity and hard constraints are violated, and both enter solely to exclude nonlinear harmonics (Props. 2, 4; Thm. 2).

AuthorsOpenReview ↗

Response to Reviewer 1KAq (2/2)

Breaking one assumption at a time degrades AP gradually and keeps it well above LeJEPA. Starting from the synthetic setup of Table 9 (ρ = (0.95, 0.50)), we break one condition at a time while keeping the scored factors fixed: heavy-tailed t₃ noise (Gaussianity), upper-triangular coupling [[0.95, c], [0, 0.50]] (diagonal dynamics), and rotational coupling [[0.95, −c], [c, 0.50]] (reversibility). MCC across 5 seeds:

Assumption violated in the dataLeJEPAAP+SIGRegAP+PairSIGReg
none (all assumptions hold).937.995.997
Gaussianity: heavy-tailed t₃ noise.907.938.934
diagonal dynamics: coupling c = 0.1.929.989.992
diagonal dynamics: coupling c = 0.3.798.957.982
reversibility: rotation c = 0.1.942.995.997
reversibility: rotation c = 0.3.956.990.993

Lighter non-Gaussian noise (t₅, Laplace, uniform) keeps AP above .987, and breaking reversibility has almost no effect.

The theory's error bounds hold on real data, and PairSIGReg works effectively as a soft regularizer. On Reacher, Prop. 8's perturbation bound tracks the small measured wrist error (0.029 vs. bound 0.19, Table 23). In Figure 3(a), the data follow Eq. 5 by construction, and the soft PairSIGReg penalty suppresses the excluded second-order harmonic (share ≤ .003).

On joint Gaussianity, physical factors are bounded angles or discrete grids, so a representation that decodes them linearly cannot be strictly Gaussian. Gaussianity instead serves as a regularizing target (as in SIGReg): SIGReg pulls marginals closer to Gaussian (|skewness| ≤ .22, |excess kurtosis| .07–.53 vs. −0.8 to −1.5 for the factors), and PairSIGReg's joint penalty prevents subspace collapse, lifting MPI3D-real worst-factor R² from .070 to .893 (Table 12). We have added these diagnostics to Appendix D.8 (Table 24).

W3 / Q3: How does post-hoc AP relate to AMUSE and what is new beyond it, and does the method work with natural temporal transitions rather than constructed pairs?

Our framework contains AMUSE as one component; on fixed features our closed-form basis step is computationally equivalent to AMUSE, while the rest of the framework addresses representation learning. Classical AMUSE/SOBI assumes a fixed linear mixture whose span already contains the sources, then rotates it using lag covariance. For a nonlinear JEPA encoder learned from pixels, the main bottleneck is ensuring its span retains the factors (analyzed in Props. 2, 4 and guaranteed globally by PairSIGReg in Thm. 2) and aligning them cleanly (derived from the same cross-talk term for both training-time and post-hoc alignment in Thm. 1, with perturbation bounds in Prop. 8).

Empirically, a post-hoc AMUSE rotation cannot fix a broken subspace or tied timescales. Applied to Reacher baselines whose spans lose factor information (VICReg, BYOL, Barlow Twins, LeJEPA), two-label shoulder error stays above 77°, whereas AP+PairSIGReg reaches 8.9° (Table 20). On Falcor3D's tied camera axes, post-hoc rotation has no spectral gap and drops MCC to .756, whereas training-time AP reaches .912. We have made this relation to AMUSE explicit in the abstract and §2 (alongside Appendix B.7).

The method uses natural temporal transitions whenever the dataset has a time axis. OU-Spiral, Reacher, Cartpole, and the LeWM tasks pair frames a fixed number of steps apart along trajectories; constructed pairs are used only on static image grids (MPI3D, Falcor3D, Isaac3D) with no time axis. Applying the basis step to frozen I-JEPA and V-JEPA on native DeepMind Control videos paired solely by frame lag improves coordinate alignment across all six settings (e.g., .499 to .762 for I-JEPA and .453 to .700 for V-JEPA on Reacher) with full R² unchanged, while shuffling frames in time removes the gain. We have clarified the pairing setup in §5 and added these natural-video results to Appendix D.4.2 (Table 18).

Dear authors,

I appreciate your efforts to address my concerns, particularly the new control experiments and the clarification regarding AMUSE. Thank you for these additions.

However, I still have concerns about both the practical utility and novelty of the proposed approach. Specifically, I am not fully convinced by your argument that "AMUSE cannot fix a broken subspace." From Table 26 of the revised paper, standard LeWM already achieves near-perfect linear readout (R² = .999/.996 on Reacher/Two-Room), which would suggest that the relevant factors are already preserved in the learned representations, even if they are mixed across coordinates.

I find that it is unclear why post-hoc AMUSE would not be sufficient to recover the individual factors (with the benefit of not modifying the training procedure). This is why I believe that an important baseline is standard LeWM + post-hoc AMUSE, evaluated on the same downstream control tasks. Without this comparison, it is difficult to establish whether training-time AP offers any meaningful advantage over existing methods.

AuthorsOpenReview ↗

Follow up Response to Reviewer 1KAq

DDear Reviewer 1KAq,

Thank you for the follow-up and for acknowledging the new controls. AP-JEPA targets the rotation barrier: retaining the factors does not mean separating them across coordinates. Standard LeWM illustrates this distinction. On Reacher, its full R² is .999, but its native MCC is only .443. High full R² therefore does not show that our target problem is already solved. A suitable post-hoc rotation can resolve it, but identifying and justifying that rotation is itself part of our contribution. The post-hoc procedure in this comparison is the second mode of AP-JEPA, explicitly described in our abstract. Its higher MCC on these two tasks supports your intuition about when retraining is unnecessary; it does not establish that AP-JEPA is unnecessary.

AMUSE is classical, but its existence alone does not establish that it applies to learned JEPA representations. To our knowledge, no prior work had established this application or its validity conditions. We derive the connection from the JEPA objective: Theorem 1 isolates the cross-talk term responsible for basis selection, and Proposition 7 obtains the frozen-feature solution by minimizing that same term. Under the corresponding assumptions, this solution is computationally equivalent to AMUSE. Recovering a classical result under matching assumptions grounds our theory in established separation theory; it does not make the JEPA analysis redundant. Nor is this connection interchangeable with generic post-hoc processing: PCA whitening and FastICA yield substantially weaker alignment in our controls. Direct application of AMUSE can also fail when the required subspace, noise, or temporal conditions are not met. Our propositions clarify these conditions. The contribution is thus a theoretical foundation for JEPA representation identification that recovers a classical separation method as a special case.

The requested LeWM comparison reports MCC and goal-image planning success with the official planner:

Standard LeWM+ post-hoc mode+ training mode
Reacher, MCC.443.872.772
Reacher, planning86.4%91.6%90.0%
Two-Room, MCC.399.846.795
Two-Room, planning87.0%87.0%100%

The post-hoc mode uses the full map of Proposition 7 as the planning metric. Two-Room post-hoc planning uses evaluation seed 42; the other planning results use 250 episodes per training seed.

Both modes substantially improve alignment over standard LeWM, but they can change different properties of the representation. An invertible linear post-hoc map preserves the learned span and full linear R². It can separate retained factors, but cannot recover factors absent from that span. High R² on Reacher and Two-Room makes them favorable cases for post-hoc alignment. It does not contradict the limitation that rotation cannot repair an incomplete span. Training with PairSIGReg retains the most complete factor span among our evaluated variants, and training can also better suppress noise during learning. These benefits go beyond reorganizing the factors a frozen model already contains. On Two-Room, training raises planning success from 87.0% to 100%, while post-hoc alignment leaves it at 87.0% despite achieving higher MCC.

Training also matters when a downstream policy or planner uses a representation that continues to evolve, as in online RL or continual learning. A post-hoc map fitted to one checkpoint may become outdated as the encoder changes; training-time alignment acts throughout learning. In our experiments, the trained model also works directly with the official planner, whereas the post-hoc procedure requires additionally selecting the slowest kk dimensions. Our claim is not that training always beats post-hoc alignment. AP-JEPA provides both the rotation needed to align a suitable frozen representation and an objective that shapes the representation during learning. Comparing these two modes clarifies when each is useful, while standard LeWM remains their shared baseline.

Reviewer_g41gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents and develops a solution to the "rotation barrier" problem. The rotation barrier problem emerges from the the rotational invariance of the current dense prediction loss and isotropic regularizers. Addressing the rotation problem through a new anisotropic predictor that can be decomposed into a rotationally-invariant dense predictor term and a cross-talk term. Through this modification, the authors aim to learn canonical bases for the representations where each coordinate captures a single physical factor rather than a mixture of factors. Across multiple datasets they demonstrate the ability to identify coordinates with factors. Experimental results suggest that the proposed regularizers and anisotropy terms address the rotation barrier and improve coordinate-factor identifiability.

Critical strengths

The paper addresses a well-motivated limitation/failure mode of existing JEPA architectures and provides the mathematical grounding for their proposed solution and its relation to existing techniques.

The theoretical results, assuming they are accurate without having reviewed in detail the full appendices, appear fairly tight in the included results.

Compatibility with existing JEPA architectures in addition to utility when training from scratch, makes the approach flexible and multifunctional while improving interpretability. The improved interpretability does not come at the expense of linear readout.

Critical weaknesses

A key omission from the paper is the impact of the anisotropy on training time or computational complexity. While there are reported performance improvements in terms of identifiability and MCC, the gains may be more modest relative to added resource requirements for training.

Outside of one example of the Reacher dataset, all of the other datasets are left to the reader to look up and understand. Extensive descriptions may not be reasonable given the page limits but without any additional context it makes it hard to fully interpret the main results tables. The paper doesn't share the number of factors present in each dataset. As written, within the main body of the test there is insufficient information to reproduce the experimental results. Additional information is provided in the appendices but without the additional information, again, it makes it difficult to truly contextualize the results and understand the broader impact of the results.

MCC and R2R^2 are reported in the tables as percentages (and in the abstract) which is unclear and raises concerns. If the experiments were reporting the percentage of factors associated with a single coordinate through some post-processing classification assignment, that information is missing.

A key concern for the reviewer is the ability of the paper to standalone without the ~20 pages of supplementary information provided in the appendix.

Questions for authors

Please clarify why you report MCC and R2R^2 as percentages?

It was unclear to the reviewer how timescales factor in. Please clarify the

How did training times change relative to LeJEPA? What additional resources and/or training time was required to perform the post-hoc coordinate recovery?

The reviewer interpreted the distinct lag coefficients for each factor as an assumption of distinct timescales which enable the diagonal formulation. Is this correct? If so, can you provide additional justification why the distinct timescale assumption for the factors (in the theoretical results) is likely to hold in most real-world datasets?

2 comments
AuthorsOpenReview ↗

Response to Reviewer g41g (1/2)

We thank the reviewer for the careful reading and for finding the problem well motivated, the theory tight, and the method flexible for both new and pretrained JEPAs. The comments on readability led us to reorganize §5 so the main text stands on its own, and the questions on cost and timescales to a more complete analysis.

W2 / W4: Can the main text describe the datasets and factors well enough to read Tables 2–4 without the appendix?

We apologize: the order in which the paper presents its setup made the main text hard to follow on its own. We have made the following changes in the revised PDF (in blue):

  1. §5 opens with a compact table of the benchmarks (Table 2, reproduced below).
  2. §5 now states the pairing, evaluation split and seeds needed to read Tables 3–5, and points to the exact subsections of Appendix C.1–C.3 for the rest.
  3. The Metrics paragraph defines MCC and R² and their scale (W3 below).
  4. §3 explains how timescales enter and how to check them without labels (Q2 / Q4 below).
  5. §5 states the training cost, with the full cost table in a new appendix section (Appendix D.11, Table 28).

The appendix keeps the proofs and the complete hyperparameters, and each pointer from the main text names the exact appendix section.

DatasetSettingScored factors
OU-SpiralSynthetic 2D dynamics2 (latent factors)
Reacher (DMC)Pixel control2 (wrist, shoulder)
Cartpole (DMC)Pixel control2 (cart, pole)
MPI3D (3 versions)Rendered 3D images2 (arm axes)
Falcor3DRendered 3D images3 (camera axes)
Isaac3DRendered 3D images6 (robot, camera, object, lighting)

Angles are scored by their (cos, sin) columns. Labels are used only for scoring, never for training.

W3 / Q1: Why are MCC and R² shown as percentages, and are they proportions of factors?

They are not proportions of factors: they are the standard MCC and R², which lie in [0, 1]. We apologize for the confusion: we multiplied them by 100 to narrow the table columns. The revision reports them as decimals (e.g., .947 instead of 94.7) in every table and in the abstract, with adjusted column widths. MCC matches factors to coordinates one-to-one with the Hungarian algorithm on held-out data and averages the absolute Pearson correlations; the matching is part of the evaluation only, and the method never uses it. Full R² averages, over factors, the R² of a ridge probe on all coordinates, so a high full R² with a low MCC is exactly the rotation barrier. The Metrics paragraph of §5 now states these definitions.

W1 / Q3: How much training time and memory does AP add over LeJEPA, and what does post-hoc recovery cost?

The added cost is small relative to the gain. With the paper's trainer on Reacher (d = 32, batch 256, one A100, 3 runs):

MethodReacher MCC (Table 3)Time per epoch
LeJEPA.3472.89 s
dense+SIGReg.5623.33 s
AP+SIGReg.8883.07 s
AP+PairSIGReg.9473.76 s

Peak memory (3,166 MiB) and checkpoint size are identical across all four methods because the predictors are closed-form per batch and store no parameters. The diagonal predictor is cheaper than a dense one, and the gap grows with width: at d = 2048 and batch 1024, the loss takes 2.0 ms for AP+SIGReg, 9.8 ms for AP+PairSIGReg, and 18.9 ms for dense+SIGReg. Post-hoc recovery needs no training: after one forward pass over unlabeled frames, it computes two covariances and two eigendecompositions (whitening and lag) and applies the map, taking 10.6 s in total for 100,000 synthetic 1280-d features (the width of I-JEPA ViT-H) and storing one d × d matrix (6.5 MB).

AuthorsOpenReview ↗

Response to Reviewer g41g (2/2)

Q2 / Q4: How do timescales enter, is a distinct timescale per factor required, and does it hold on real data?

The diagonal predictor itself requires no timescale assumption; distinct timescales are what ensure it recovers one factor per coordinate rather than a shared block.

A factor's timescale is how fast it changes, measured by its lag correlation ρ between the two frames of a pair; a slow factor has ρ close to 1. The diagonal predictor predicts each coordinate from its own previous value with a single coefficient. A coordinate that holds one factor is predicted well by one coefficient. A coordinate that mixes a slow and a fast factor is not, because no single coefficient fits both. For example, with ρ = .9 and .3, two coordinates that each hold one factor have prediction errors .19 and .91 (total 1.10), while two 45° mixtures have .64 each (total 1.28). Training therefore pushes each coordinate to a single timescale, which aligns it with a factor (Eq. 4), and the learned coefficients converge to the factors' ρ (Prop. 1): on Reacher, the learned wrist coefficient is about .76, against a wrist ρ of .746 (Table 22).

When factors share a timescale, mixing them costs nothing, so they are recovered together as a block, while factors with different timescales are still separated (Prop. 1, Thm. 2). Which factors each learned coordinate picks up (up to order and sign; * = nonzero):

 Distinct timescales              Tied timescales (rho_3 = rho_4)
          f1  f2  f3  f4                   f1  f2  f3  f4
coord 1 [  *   0   0   0 ]        coord 1 [  *   0   0   0 ]
coord 2 [  0   *   0   0 ]        coord 2 [  0   *   0   0 ]
coord 3 [  0   0   *   0 ]        coord 3 [  0   0   *   * ]
coord 4 [  0   0   0   * ]        coord 4 [  0   0   *   * ]
 one factor per coordinate          f3, f4 share a block

Reacher's shoulder angle, encoded as (cos, sin), is such a block, and Appendix C.5 fixes the rotation inside it with two labels.

Whether timescales differ can be checked without labels from the lag spectrum of the representation; Table 22 reports the factors' ρ for every benchmark. On the native-dynamics tasks they differ (on Reacher, ρ = .75 for the wrist and .26 for the shoulder). On the constructed MPI3D and Falcor3D pairs of Table 4, every scored factor is stepped by the same rule and shares one ρ, so the theory guarantees them only as a block, separated from factors at other timescales.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank Reviewers g41g, 1KAq, and MCQE for their constructive reviews. We are encouraged that they view the rotation barrier as a well-motivated problem (g41g, 1KAq, MCQE), find our subspace–basis decomposition and theory rigorous (g41g, 1KAq, MCQE), and appreciate our thorough experiments across diverse benchmarks, clear ablations, and analysis without sacrificing linear readout (g41g, MCQE).

Additional results in the responses

  1. Extending control to the LeWM world model (1KAq, MCQE A). Adding our diagonal and PairSIGReg terms to LeWM raises native MCC while keeping linear readout high, matches Reacher goal-image planning, improves Two-Room from 85% to 100% and the 3D robot-arm task Cube from 72.7% to 78.0%, and enables single-factor control from a few slowest native coordinates chosen without labels.

  2. Clarifying the predictor and PairSIGReg roles across all baselines (1KAq, MCQE C). Applying our basis step after training to all Tables 3–4 baselines aligns coordinates only when the span already retains the factors. Training with AP preserves and aligns factors even under tied timescales (Falcor3D), while PairSIGReg prevents subspace loss and lowers worst-factor error.

  3. Clarifying assumption sensitivity and optimization trajectories (1KAq, MCQE B). Real benchmarks are nearly reversible and diagonal, violating only Gaussianity; breaking assumptions individually degrades AP gradually while staying above LeJEPA.

  4. Clarifying natural video pairing and computational cost (g41g, 1KAq, MCQE D). On native DMC videos paired purely by time intervals, our basis step improves frozen I-JEPA and V-JEPA across all six settings. On Reacher, AP+SIGReg takes 3.07 s per epoch vs. 2.89 s for LeJEPA at identical peak memory; post-hoc takes 10.6 s on 100k synthetic 1280-d features.

Changes in the revision (marked in blue)

We have uploaded a revised PDF in which every added or rewritten passage is marked in blue, and the main text still fits in 10 pages. Table and figure numbers in our responses refer to the revised PDF; the question headers keep the reviewers' original numbering.

  • Abstract, §1, §2: the relation to AMUSE (our framework contains AMUSE as one component; on fixed features our closed-form basis step is computationally equivalent to it), the coordinate–orthogonal–linear identifiability hierarchy (§1), and related work on linear identifiability in meta-learning, LIFS, Koopman embeddings, time-contrastive learning and SlowVAE (§2, Appendix A).
  • §3: the timescale condition and how to check it from lag spectra without labels.
  • §5: a benchmark table (Table 2), the pairing and evaluation setup with pointers to Appendix C.1–C.3, MCC and R² on the 0–1 scale in every table, the bolding rule, the training cost, and the LeWM planning results (Table 6).
  • Appendix: the full post-hoc comparison (D.4.1, Table 17), natural-video results (D.4.2, Table 18), whitening trajectories (D.1.2, Figure 5), assumption diagnostics (D.8, Table 24), LeWM details with planning rollouts (D.10, Tables 26–27, Figure 8), and the cost table (D.11, Table 28).
AuthorsOpenReview ↗

General Response to the Follow-up Discussion

Dear AC and Reviewers,

Thank you for the continued discussion, we truly appreciate your engagement!! Since much of the follow-up has focused on post-hoc alignment, we would like to restate in one place what AP-JEPA is about and where the post-hoc step fits.

AP-JEPA addresses the gap between retaining factors and recovering them at the coordinate level. A JEPA may retain all target factors and still mix them across coordinates. It may also retain only part of them. Our framework distinguishes these cases and explains what is needed for coordinate-level recovery.

The post-hoc procedure discussed in the follow-ups is part of AP-JEPA, explicitly described in our abstract. AMUSE is classical, but to our knowledge, no prior work had established its application to JEPA world-model representations or the conditions under which it works. Our theoretical derivation led to a computationally equivalent procedure. Recovering a classical result under matching assumptions grounds our framework in established theory; it does not make the new connection redundant. The missing contribution was not another separation algorithm. It was establishing why and when such an algorithm can identify factors in learned JEPA representations. Our analysis provides that foundation. Successful post-hoc alignment therefore supports our solution to the rotation barrier, rather than showing that the problem was already solved.

Training and post-hoc alignment are two routes within this unified framework, not competing claims about which must always win. Post-hoc alignment separates factors already present in a frozen representation. Training can also shape which factors are retained and how they are represented. The two routes therefore address different settings and offer different benefits. Our experiments, including the additional downstream evaluations requested by the reviewers, demonstrate their effectiveness. Comparing the two helps establish when each is useful. It does not turn one part of our contribution into evidence against the other.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.