acceptodds
Under review as a conference paper at ICLR 2027

Lower Loss by Forgetting: How Trainable Targets Suppress Information in JEPA World Models

Abstract

Joint-embedding predictive architectures (JEPAs) learn world models by predicting future representations and can reduce prediction error by suppressing information already present in their representations. We identify gradients through the learned prediction target as a mechanism for this suppression. Our gradient decomposition shows that, with an optimal predictor and a linear encoder aligned with independent factors, the target contribution attenuates each factor at a rate proportional to its innovation variance. A quadratic covariance penalty yields a closed-form retention threshold and links retained variance to each factor's weight in the linear model's planning distance. Controlled experiments with initially decodable synthetic signals support the predicted ordering, with higher-innovation signals losing decodability earlier. In a simulated pushing task, a model acquires and subsequently suppresses information about block position. A state variable can also remain decodable while having little influence on the latent distance used for planning. On the Cube manipulation task from OGBench, weaker regularisation reduces normalised prediction error by while lowering contact-related decodability and planning success. Paired head interventions improve Cube planning, but their benefits depend on the task. These findings identify a mechanism by which predictive training can impair a representation for control, and motivate evaluating information retention and its influence on planning distance alongside prediction accuracy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, joint-embedding predictive architectures, world models, representation learning, latent-space planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yXyUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates a failure mode of latent prediction: jointly optimising an encoder and its prediction targets can make prediction easier by suppressing represented variables. It separates context and target gradients, characterises innovation-dependent attenuation under a covariance penalty, and connects retained variance to Euclidean planning distance. Signal-injection and pushing experiments track previously decodable information during training; Cube experiments compare prediction error, contact-variable decodability and planning under regularisation changes and head interventions. The central distinction is between information being recoverable and that information meaningfully influencing the planner’s cost.

Critical strengths
  1. The controlled experiments distinguish suppression from failure to acquire information. Section 3.1 starts with decodable independent signals and compares matched training conditions with and without target gradients. The loss of higher-innovation signals repeats across LeWM and DINOv2 heads; nonlinear probes corroborate the principal result. This is substantially stronger evidence for active suppression than finding low endpoint probe accuracy alone. The training trajectories additionally test the predicted ordering of suppression rates.

  2. The distinction between decodability and planning-distance sensitivity is experimentally substantive. In Section 3.2, an inverse-dynamics latent supports block decoding at approximately R2=0.85R^2=0.85, yet responds weakly to block displacement relative to pusher displacement. The controlled scene perturbations make this distinction concrete rather than inferring geometry from probe scores. Table 2’s paired Cube improvement with target detachment (5.35.3 percentage points, with a start-bootstrap interval of [3.0,7.8][3.0,7.8]) adds practical relevance to the diagnostic argument.

Critical weaknesses
  1. Much of the target-gradient mechanism has a close theoretical precedent. Ni et al. (2024), Appendix B, Equations 88 and 101, already identify the conditional-variance contribution to self-prediction error and the additional online-target gradient that can remain nonzero at predictive sufficiency. Consequently, identifying this extra gradient and contrasting it with stop-gradient training is not itself a new mechanism. arXiv The manuscript acknowledges these predecessors, but the clearer increment is the factor-specific attenuation and regularisation analysis, together with the controlled measurements. This limits the weight of the theory as an independently novel contribution, without negating the empirical contribution.

  2. The mechanism only partly explains the benchmark planning results. Cube’s measured innovation does not predict which variables are suppressed, and the weak-regulariser runs reach their highest measured retention at the final checkpoint rather than exhibiting the anticipated final-versus-peak decline. Thus, Cube establishes a cross-model loss–retention trade-off, not innovation-ordered temporal forgetting. Moreover, head retraining changes the predictor and representation geometry together, and Section 4 reports planning differences at similar decodability. The intervention therefore establishes an effect of the training choice, but not that contact-information retention mediates the planning gain. These limitations are disclosed appropriately, yet materially restrict the explanation of benchmark control performance.

Questions for authors
  • Beyond the factor-specific rate and regularisation threshold, which aspect of Proposition 1 provides a substantive extension of Ni et al.’s conditional-variance and online-target-gradient analysis?

  • For the paired Cube heads, how do their goal-cost rankings compare when scoring identical encoded simulator outcomes, particularly around contact events? Such a fixed-outcome comparison would help distinguish changes in cost geometry from improvements in rollout prediction, without requiring another training study.

3 comments
AuthorsOpenReview ↗

Response to Reviewer yXyU (Part 1) - Ni et al. (2024) [W1, Q1] and what Cube establishes [W2]

Due to the character limit, we have divided our response into two consecutive comments. This is Part 1 of 2.

Thank you for a careful and accurate review. Your summary of the central distinction, between information being recoverable and that information influencing the planner's cost, is the reading we intend.

W1 / Q1 — Prop. 1 and Ni et al. (2024). We agree. The conditional-variance term and the extra online-target gradient at predictive sufficiency are in Ni et al. (Appendix B, Eqs. 88 and 101), and Prop. 1 taken by itself is not a substantive extension. It adds only bookkeeping that the later results use: it names the two residuals (the latent innovation and the sufficiency gap), and it reads the target term as a functional gradient that moves each target embedding towards the average prediction made for it. We will cite Eqs. 88 and 101 at Prop. 1, label it a restatement in our notation, and order the contributions accordingly. The extensions are in the results that follow it:

  • Prop. 2 (removal, not only non-acquisition). For a general nonlinear encoder, adding any amount τ\tau of an independent signal to a predictively sufficient representation raises the optimal prediction error by at least τ2Ik/D\tau^2 I_k/D (IkI_k: the signal's innovation variance; DD: the latent dimension), while with the target held fixed the same signal costs nothing. This turns "an extra gradient exists" into a statement about content that is already represented, which §3.1 tests.
  • Prop. 3 (the regulariser). Ni et al. analyse objectives without an anti-collapse regulariser. A smooth moment-based regulariser (SIGReg, VICReg) supplies no first-order restoring force for an independent signal, so it does not counteract the extra term.
  • Prop. 5 and Cor. 1. The closed-form retained variance vi⋆=(1−qi/2λ)+v_i^\star=(1-q_i/2\lambda)_+ of a factor with innovation fraction qiq_i at regulariser weight λ\lambda, the resulting loss–retention trade-off, and the identification of retained variance with a factor's weight in the planning distance.

W2 — what Cube establishes. We agree with both statements and will use your formulation in the abstract, §4 and the discussion: Cube establishes a cross-model loss–retention trade-off, not innovation-ordered temporal forgetting, and the head interventions establish an effect of the training choice on planning, not mediation by contact-information retention. Three new analyses sharpen this without changing it.

  1. Feature directions. The theory orders independent factors; the Cube state variables are coupled (gripper opening, contact and cube height change together at a grasp), and their estimated innovation fractions are small and close together (at most 0.105; contact flag 0.067). So we tested the ordering on directions of the representation. Over 177 whitened directions of the three weak-regulariser (λ=0.0117\lambda=0.0117) backbones (image encoders), the latent variance an end-to-end head allocates to a direction, relative to its stop-gradient partner, decreases with the direction's innovation. The caveat: in these backbones, innovation and feature variance are themselves strongly related (ρ≈−0.95\rho\approx-0.95), and with log variance controlled the partial correlation is only −0.16 [−0.34, −0.01]. The clean test of the ordering therefore remains §3.1, where all signals have equal variance.
  2. Forgetting within a run, at the head level. We retrained the paired heads on the fixed backbones with a probe every 500 steps. End-to-end heads fall 0.07–0.18 below their initial contact-related decodability, and on two of the three backbones they end 0.13–0.14 below it. Stop-gradient heads dip by at most 0.02 and end 0.07–0.12 above it. On a fixed backbone this is a loss within a run, and it occurs only with target gradients. Our statement about full models is unchanged.
  3. A full-model intervention. We trained LeWM from pixels with a detached target at the weak weight, using the seed, split and settings of one end-to-end model, with nothing tuned. Contact-related R2R^2 is 0.77 (end to end: 0.52–0.67; gap between backbone and latent probes 0.13 against 0.17–0.26), at a much higher one-step error (40×10−340\times10^{-3} against (1.3(1.3–1.5)×10−31.5)\times10^{-3}). Planning is 57.7% against 52.7% (+5.0 points [0.0, 10.0]), not distinguishable from zero. At the default weight (λ=0.09\lambda=0.09), where end-to-end models lose little after the backbone (gap 0.02–0.03), the same intervention instead lowers what the backbone itself acquires (backbone probe R2R^2 0.85 against 0.94) and lowers planning (−9.0 [−14.0, −4.3]). In full-model training, detaching the target is therefore not a remedy by itself. This supports your reading: the benchmark results show an effect of the training choice and do not identify a single mediator.
AuthorsOpenReview ↗

Response to Reviewer yXyU (Part 2) - fixed-outcome comparison of goal costs [Q2]

Due to the character limit, we have divided our response into two consecutive comments. This is Part 2 of 2.

Thank you for this suggestion. It separates cost geometry from rollout accuracy without new training, and we have run it.

Design. We used 600 origins: the 300 planning starts, plus 300 states from expert episodes, each within five model steps before the gripper makes or breaks contact. From each restored simulator state we execute 16 candidate action sequences that do not depend on any model: the expert continuation and 15 perturbed copies. Each final frame is encoded once by the frozen backbone that the two heads of a pair share, and each head scores it by its latent distance to its own encoding of the goal. There are six pairs (three weak-regulariser backbones, two initialisations); intervals are 95% bootstrap intervals over origins.

Kendall τ\tau between two headsAll originsContact changeNo contact change
End-to-end vs. stop-gradient0.630.590.77
Two initialisations, same objective0.900.890.94
Difference−0.27 [−0.29, −0.26]−0.30 [−0.32, −0.28]−0.17 [−0.20, −0.15]

The training choice changes the cost geometry, most around contact. On identical encoded outcomes the paired heads disagree far more than two initialisations of one objective, and most where contact changes. Both heads score the same frames through the same backbone, so this cannot come from the predictor.

Neither geometry agrees measurably better with task progress. Rank agreement (Spearman, per origin) with the final cube-to-goal distance is 0.171 for stop-gradient heads and 0.163 end to end (difference +0.008 [−0.010, +0.027]). The interval includes zero at contact changes and elsewhere as well. The stop-gradient heads' top-ranked candidate ends slightly closer to the goal (3.50 against 3.80 cm behind the best candidate; −0.30 [−0.53, −0.09]). With each head's own predicted outcomes the difference disappears (−0.11 [−0.29, +0.07]).

Nor are the stop-gradient rollouts more accurate. Their one-step error is three to four times higher, and across the 20 retrained heads one-step error correlates positively with planning success (ρ=+0.75\rho=+0.75).

Conclusion. The paired heads differ in the geometry of the goal cost, mainly around contact. We do not attribute the +5.3-point planning gain to it: a fixed set of 16 candidates does not probe closed-loop search, where the gain may arise. This matches your reading under W2. We will state it in §4 and report the comparison in the appendix.

The push world (§3.2), from existing records. Of the 166 scripted pushes that bring the block to its goal, the encoded true final frame is closer to the goal than the start frame in 83–89% of cases with stop-gradient models, 72–76% end to end and 70–73% with inverse dynamics (three models each). With each model's own rollouts, the figures are 71–78%, 74–79% and 72–73%. Here too, the stop-gradient latent has the more informative cost on true outcomes, and the advantage disappears under the models' own rollouts.

AuthorsOpenReview ↗

Revised PDF

We have uploaded a revised PDF, with all changes marked in blue. Following your review, it credits Ni et al. (2024, Eqs. 88 and 101) at Prop. 1, adopts your formulation of what the Cube results and the head interventions establish, and adds the fixed-outcome comparison to §4 and Appendix G.

Reviewer_87KAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies how learning a predictor and its representation can suppress information in JEPA-style world models. The authors decompose the encoder gradient into context and target contributions and, under linearity assumptions, derive theoretical results. Experiments with appended synthetic signals and manipulation benchmarks examine information retention, latent-distance sensitivity, and planning performance.

Critical strengths

The gradient decomposition and linear retention results could give a concrete account of how predictive training can suppress represented information. This type of theory could prove to be informative about JEPA style models.

Initially synthetic signals with varying innovation rates provide a direct test of the proposed mechanism and allow for empirical tests.

Distinguishing decodability from influence on latent distance helps explain why recoverable information may contribute little to planning.

Critical weaknesses

Despite my familiarity with world models and JEPAs, I really struggled to identify the precise claim of the paper and understand how the propositions and experiments support it.

Synthetic experiments might support the mechanism, but the connection doesn’t appear obvious.

Several figures, particularly Figures 2–3, have small text, crowded legends, and overlapping annotations that obscure the comparisons.

Proofs written as written are presented in a nonstandard (and perhaps AI generated?) way that makes them very difficult to check. Several parts of the experimental setup are confusing. Appendix B makes protocols difficult to reconstruct. A table specifying frozen components, gradient paths, checkpoint selection, splits, and seed counts would help. Appendix D is also unusual and an odd inclusion in a machine learning paper. A concise description of supported and failed predictions would better clarify the implications.

Questions for authors
  1. What is the paper’s precise central claim, and which experiments directly test each theoretical prediction? Can this be clarified in the main text to make it more human-readable?

  2. Could the proofs be presented more explicitly, with assumptions, intermediate steps, and conclusions clearly separated, so readers can easily understand and check them, as well as asses its novelty?

  3. Several passages assume familiarity with terms, notation, and experimental choices before explaining them adequately (for instance AR(1)) Could the paper be revised to define these before use and make the exposition more self-contained?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 87KA (Part 1) - the central claim, and which experiment tests which prediction [Q1]

Due to the character limit, we have divided our response into two consecutive comments. This is Part 1 of 2.

Thank you for the review and for the concrete suggestions on presentation.

Q1 — the central claim. When the prediction target of a JEPA world model is produced by the same encoder that is being trained, the gradient through the target lowers the prediction loss by removing information the representation already holds, the least predictable content first. The regulariser that prevents collapse does not counteract this, and content removed in this way stops influencing the latent distance that the planner minimises. The paper analyses this mechanism in an existing class of models (JEPA world models trained end to end, with the gradient also passing through the target, such as LeWM); it does not propose a new model.

Terms used below. The backbone is the image encoder; the head is everything after it (projector, predictor, action encoder), and its output is the latent, in which the model predicts and the planner measures distance. The target gradient is the part of the prediction-loss gradient that reaches the encoder through the predicted embedding; a detached (stop-gradient) target removes it. The innovation of a signal is the part of its next value that cannot be predicted from its past and the actions; q∈[0,1]q\in[0,1] is its share of the signal's variance (q=0q=0: fully predictable; q=1q=1: white noise). λ\lambda is the regulariser weight. A variable is decodable if a linear probe recovers it from the latent on held-out episodes; we report the probe's R2R^2. Cube is the OGBench task in which a robot arm grasps and places a cube; its contact-related variables are gripper opening, contact flag and cube height (mean R2R^2). §3.2 is a simulated pushing task, learned from pixels, in which a pusher moves a block.

PredictionFromTestOutcome
1. A represented, independent signal is removed with the target gradient and kept with a detached targetProps. 1–2§3.1: signals appended to fixed backbone features; the two conditions differ only in the target gradientSupported. Signals with q≥0.05q\ge0.05: R2≤0.00R^2\le0.00 with the target gradient, 0.94–0.99 detached; same on DINOv2
2. Less predictable signals are removed fasterProp. 1§3.1, during trainingSupported. Rank correlation between qq and the time to halve decodability: −0.97 and −0.93 (two seeds)
3. A larger regulariser weight keeps signals of higher innovationProps. 3, 5§3.1, three weightsOrdering supported. The exact cutoff q=2λq=2\lambda holds only in the linear-Gaussian model of §2
4. Weaker regularisation lowers prediction error and retention togetherProp. 5§4: 13 Cube models at eight weightsSupported across models (3.4× lower one-step prediction error; contact-related R2R^2 0.91–0.92 → 0.52–0.67). Not as an ordering of individual Cube variables by innovation
5. Decodable content can have little weight in the planning distanceCor. 1§3.2: one object displacedSupported. The block is decodable (R2=0.85R^2=0.85), yet moving it changes the latent distance by only 0.3–1% as much as an equal move of the pusher
6. Suppression occurs during training from pixelsProps. 1–2§3.2, during trainingSupported for the block (R2R^2 0.96 after 1,000 steps, 0.62 at the end)
7. Protecting represented content changes planningCor. 1§4: heads retrained on fixed Cube backbones, with or without the target gradientSupported on Cube (+5.3 points of planning success [3.0, 7.8]); not on Push-T

How the synthetic experiment connects to the theory. Props. 1–2 concern a signal that (i) the representation already contains, (ii) is independent of everything else, and (iii) has a known innovation. In a benchmark none of these is under control: one cannot tell "never learned" from "learned and removed", physical variables are coupled, and innovation must be estimated. The appended-signal experiment constructs a case in which all three hold by design. We append extra channels to the backbone features of each frame. Each carries a random signal generated independently of the images as a first-order autoregressive process, AR(1): xk+1=1−q xk+q ϵkx_{k+1}=\sqrt{1-q}\,x_k+\sqrt{q}\,\epsilon_k, with independent standard-normal ϵk\epsilon_k. Each value is the previous one, shrunk, plus fresh noise, so the signal has unit variance and its innovation share is exactly the qq we choose. The signals are decodable before training. A new head is then trained with or without the target gradient. The experiment thus realises the assumptions of the propositions; §3.2 then drops (i), and §4 drops all three.

We will state the claim in this form in the introduction and add the table to the paper.

AuthorsOpenReview ↗

Response to Reviewer 87KA (2/2) - proofs, protocols, Appendix D, figures and terms [Q2, Q3]

Due to the character limit, we have divided our response into two consecutive comments. This is Part 2 of 2.

Q2 — the proofs. We would be grateful if you could tell us what you found non-standard, and which proof or step was hard to check. Appendix A is written compactly for space, and some intermediate steps are shortened. We are glad to write out every step of any proof you point to, here in the discussion, in whatever layout you consider standard (assumptions, numbered steps, conclusion).

For reference, the proofs use only standard tools:

  • the conditional mean as the least-squares predictor, and the tower property of conditional expectation (Props. 1–2);
  • the conditional Jensen inequality (Prop. 2b);
  • differentiation under the expectation (Prop. 3);
  • Ky Fan's minimum principle for sums of eigenvalues (Prop. 4);
  • von Neumann's trace inequality (Prop. 5).

On novelty: Prop. 1 restates Ni et al. (2024, Eqs. 88 and 101) in our notation, and Prop. 4 is classical (predictable feature analysis; Richthofer & Wiskott, 2015). Props. 2, 3(b–c) and 5 are new in this form to our knowledge, and Cor. 1 follows from Prop. 5.

Our use of AI tools is disclosed in the paper's AI-use statement, and we take full responsibility for every statement and proof.

Appendix B — protocols. We will add the table you describe. In short:

ExperimentTrainedFrozenTarget gradientCheckpointHeld outRuns
Appended signals (§3.1)new headbackboneon vs. detachedlowest validation lossdisjoint probe episodes3 heads per condition on the LeWM-Cube and DINOv2 backbones; 1 on each other backbone
Pushing task (§3.2)whole model, from pixels—on; detached; on + inverse dynamicsfinal, after 40k steps (fixed in advance)400 of 4,000 episodes3 per objective
Benchmark models (§4)whole LeWM, from pixels—onepoch 10Cube: 10% of episodes22 Cube, 3 Push-T
Head interventions (§4)new headbackbone of a benchmark modelon vs. detachedlowest validation lossas for the backbone20 heads

Inverse dynamics is an added loss that predicts the actions from two consecutive latents. Planning comparisons use the same fixed start states for every model (300 on Cube, 200 on Push-T).

Appendix D. We wrote down each prediction, with a numerical criterion, before evaluating it, and Appendix D reports every outcome, so that the failures are as visible as the successes. Of 47 predictions, 21 met their criterion, 4 met it in part, 21 did not, and 1 had not been run at submission (it has since been run, and its retention criterion is not met). The main failures, and what they imply:

  • the innovation ordering does not carry over to coupled Cube variables (rank correlation between a variable's estimated innovation and its change in decodability: +0.01, where at most −0.5 was predicted);
  • Cube shows a loss–retention trade-off across models, not a decline in decodability within a training run;
  • the largest qq that survives training depends on the backbone, so the exact cutoff q=2λq=2\lambda belongs to the linear-Gaussian model;
  • retaining more does not by itself improve planning on Push-T.

We agree that a concise summary of this kind serves the reader better than the full record, and we will replace Appendix D by it.

Figures 2–3. We will set all text to at least 8 pt, label curves directly in place of crowded legends, remove annotations that overlap data, and move secondary panels to the appendix.

Q3 — terms. We will define every term at first use, as in comment 1 above (including AR(1), which the submission defined only in Appendix B), and add a notation table. If other passages were hard to follow, we would be glad to know which, and to clarify them here.

AuthorsOpenReview ↗

Revised PDF

We have uploaded a revised PDF, with all changes marked in blue. Following your review, the central claim is stated in the introduction, terms such as AR(1) are defined at first use, Appendix B opens with a protocol table and a notation table, Appendix D opens with a summary of supported and failed predictions, and Figures 2–3 are redrawn. We would be glad to clarify any part that is still hard to follow.

Reviewer_spHROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how trainable prediction targets in JEPA world models can suppress information that is already represented in the latent space. The authors provide a theoretical analysis of the target and context gradients, showing that factors with higher innovation variance can be suppressed more strongly. They further connect information retention to latent distance used for planning. Experiments on controlled signals, a simulated pushing environment, and OGBench tasks provide evidence for this phenomenon.

Critical strengths
  1. Important problem and interesting insight. The paper studies an important problem in JEPA world models: minimizing prediction loss may lead to the suppression of information that is useful for downstream planning. I find this observation intuitive and potentially important for future research on latent world models.

  2. Solid theoretical analysis. The gradient decomposition and the analysis of information retention provide a clear theoretical explanation of the proposed phenomenon. I appreciate the effort to connect the training objective with representation learning and planning behavior.

  3. Useful discussion of representation quality. The paper highlights that lower prediction error and higher state decodability do not necessarily lead to better planning performance. I think this is a useful point when evaluating world models.

Critical weaknesses
  1. Limited novelty and theoretical insights. Although I find the main idea interesting, the claimed contribution seems somewhat incremental. The general observation that predictive objectives favor predictable information is already known. The paper provides a more specific explanation through target gradients, but I am not fully convinced that this leads to significantly new insights for recent progress in JEPA or latent world models. I would like the authors to better clarify what new understanding their analysis provides beyond existing work.

  2. Experimental validation is not sufficiently convincing. This is my main concern. Most of the evidence comes from controlled synthetic signals and a relatively simple pushing environment. While these experiments help illustrate the proposed mechanism, they are too simplified to support broader conclusions about practical JEPA world models. The OGBench experiments provide some additional evidence, but their scope is still limited. I think the paper needs stronger validation on more challenging and realistic environments, especially tasks involving complex dynamics and contact-rich interactions.

  3. Limited generality of the empirical conclusions. The current experiments do not clearly establish how general the proposed phenomenon is across different world models and tasks. For example, the benefit of stop-gradient targets is task-dependent, and the observed suppression in neural models does not fully follow the theoretical predictions. More experiments with different model architectures, tasks, and training settings would help determine when this mechanism matters in practice.

I am open to increasing my rating if the authors can address these concerns with stronger experimental evidence.

Questions for authors
  1. Novelty and contribution. The preference of predictive objectives for more predictable features has been discussed in prior work [1, 2], and the learning dynamics of self-predictive representations, including the role of stop-gradient, have also been studied [3]. Could the authors clarify what new insights the proposed target-gradient analysis provides beyond these existing findings? In particular, how does this analysis improve our understanding or development of modern JEPA world models?

  2. Validation in more challenging environments. The controlled-signal and pushing experiments are relatively simple. Could the authors provide additional evidence from more realistic environments, especially manipulation tasks involving complex dynamics and contact-rich interactions? I believe such experiments would significantly strengthen the practical relevance of the findings.

  3. Generality of the proposed mechanism. The experiments show that the retention threshold depends on the encoder, and the estimated innovation of individual Cube variables does not fully explain which information is suppressed. How general is the proposed mechanism across different architectures, training objectives, and environments? Can the authors provide stronger empirical evidence supporting its generality?

[1] Richthofer, S., and Wiskott, L. Predictable Feature Analysis. ICMLA, 2015.

[2] Littwin, E., et al. How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks. NeurIPS, 2024.

[3] Tang, Y., et al. Understanding Self-Predictive Learning for Reinforcement Learning. ICML, 2023.

4 comments
AuthorsOpenReview ↗

Response to Reviewer spHR (Part 1) - what is new [W1, Q1], and new full-scale evidence on contact-rich manipulation [W2, Q2]

Due to the character limit, we have divided our response into two consecutive comments. This is Part 1 of 2.

Thank you for the constructive review, and for saying what would change your assessment; the experiments it calls for are below.

Q1 — what is new beyond [1–3]. We agree that predictive objectives favour predictable features [1, 2], and that the learning dynamics of stop-gradient self-prediction have been analysed [3] (also Ni et al., 2024, who identify the extra gradient of an online target). These works describe which features are selected, or learned first. Our question is what happens to information that a representation already holds, when the target is produced by the encoder being trained and collapse is prevented by a distribution regulariser instead of a stop-gradient. This is the design of LeWM and of other JEPA world models trained end to end. Three results are new:

  • Removal, not only non-acquisition (Prop. 2). An independent signal present with amplitude τ\tau raises the best achievable prediction error by at least τ2Ik/D\tau^2 I_k/D (IkI_k: its innovation variance; DD: the latent dimension); with a fixed target it costs nothing. Signals appended to DINOv2 features, decodable before training (R2R^2 0.75–0.85), fall to R2≤0.01R^2\le0.01 with target gradients once their innovation fraction (unpredictable share of variance) is q≥0.05q\ge0.05, and stay at R2≥0.83R^2\ge0.83 without them. In the pushing task, a model trained from pixels first acquires block position (R2=0.96R^2=0.96) and then loses much of it (0.62).
  • The regulariser does not prevent it (Props. 3, 5). It supplies no first-order restoring force, and its weight λ\lambda sets what is kept: a factor retains variance vi⋆=(1−qi/2λ)+v_i^\star=(1-q_i/2\lambda)_+. Preventing collapse and retaining information are separate problems.
  • Decodable is not usable (Cor. 1). There, block position is decodable from an inverse-dynamics latent (R2=0.85R^2=0.85), yet moving the block changes the planner's latent distance by only 0.3–1% of what an equal move of the pusher does.

What this changes for JEPA world models:

  1. It identifies which designs are exposed: trained targets with a distribution regulariser, but not frozen-encoder, stop-gradient or momentum (EMA) targets.
  2. Prediction loss is not a safe criterion for model selection. On Cube, weaker regularisation gives 3.4× lower one-step prediction error and worse planning, and across 20 heads (projector, predictor, action encoder) retrained on fixed image encoders, lower error goes with lower success (ρ=+0.75\rho=+0.75).
  3. Two inexpensive diagnostics expose the problem: the gap between probes on the image encoder and on the planning latent (0.17–0.26 at the weak weight λ=0.0117\lambda=0.0117, against 0.02–0.03 at the default λ=0.09\lambda=0.09), and the response of the latent distance to a controlled displacement of one object.

Q2 — contact-rich manipulation, your main concern. You asked for stronger evidence on a contact-rich task; we have added it at full scale, with every model trained from pixels, on OGBench Cube: a 6-DoF arm must grasp, lift and place a cube, seen in 2242224^2 images. (The simple settings of §3 exist to rule out "never learned" and to set innovation by hand.)

New full-scale experiment on CubeResult
Inverse dynamics added to training, two further seeds (three in total)Planning improves in every seed: 66.7% vs. 59.7%, 68.7% vs. 58.0%, 69.3% vs. 61.7% against the default model of the same seed; mean paired gain +8.4 points [5.0, 11.9]. Contact-related R2R^2 (mean of gripper opening, contact flag, cube height) 0.974–0.975 in all three (default: 0.91–0.92)
Detached target in full-model training, at the weak weight (where end-to-end models lose most after the encoder)Contact-related R2R^2 0.77 against 0.52–0.67 end to end, at a far higher one-step error (40×10−340\times10^{-3} vs. (1.3(1.3–1.5)×10−31.5)\times10^{-3}): the predicted loss–retention trade-off, now produced by an intervention on the target path
End-to-end and stop-gradient heads scoring identical simulator outcomes (600 states, 16 action sequences each)The target path changes how the planner's cost ranks the same outcomes, most where the gripper makes or breaks contact (Kendall τ\tau 0.59 vs. 0.89 for two initialisations of one objective; 0.77 vs. 0.94 elsewhere)

With the submission's 22 Cube models and 20 retrained heads (+5.3 points [3.0, 7.8] for stop-gradient heads), the contact-rich benchmark now carries the claim at full scale: the loss–retention trade-off, across models and by intervention, and planning gains from changing the training.

AuthorsOpenReview ↗

Response to Reviewer spHR (Part 2) - realistic environments [Q2], and generality across architectures, objectives and environments [W3, Q3]

Due to the character limit, we have divided our response into two consecutive comments. This is Part 2 of 2.

Q2, continued — more realistic environments. The mechanism concerns JEPA world models trained end to end from pixels, whose prediction target is produced by the encoder being trained. To our knowledge, this class is evaluated on simulated control benchmarks: LeWM's own suite is Two-Room, Reacher, Push-T and Cube, and Cube is its most contact-rich task. JEPA world models that have been run on real robots, such as V-JEPA 2-AC, predict in the space of a frozen pretrained encoder, and video-generation world models predict pixels or the codes of a separately trained tokenizer. Their targets are fixed, so the mechanism does not arise, and a comparison with them would not test it. Cube is thus the most demanding setting in which the question can be asked of the models it concerns.

Already varied in our submission. The appended-signal test covers:

  • six encoders (LeWM on Cube at two regulariser weights and on TwoRoom, a navigation task; LeWM with inverse dynamics, DINOv2-S/14, and a third-party encoder trained with simulator-state supervision);
  • four regulariser weights;
  • three architectures (ViT with a transformer predictor; a convolutional encoder with an MLP predictor trained from pixels; a linear JEPA);
  • logged robot channels (e.g. gripper velocity) as well as synthetic signals, read with linear and nonlinear probes.

In every case, target gradients remove every signal with q≥0.35q\ge0.35, a stop-gradient target keeps all signals, and the prediction loss is 8–17× lower in the first case.

New: other objectives and a third environment (same protocol as §3.1):

VariantTrained targetNo gradient through the target
Momentum (EMA) target, momentum 0.996—All signals kept: minimum R2R^2 0.94 (LeWM Cube, three heads) and 0.82 (DINOv2); loss as with a stop-gradient, about 8× that of a trained target
VICReg instead of SIGReg (LeWM's regulariser)Signals removed: none of the nine is decodable at a base weight ww; at 4w4w, those with q≥0.2q\ge0.2 are removed and q=0.1q=0.1 in partAll kept (minimum R2R^2 0.96–0.97); loss 8–12× higher
Push-T encoder (seventh encoder, third environment)q≤0.05q\le0.05 kept (R2≥0.98R^2\ge0.98), q≥0.1q\ge0.1 removed (R2≤0.03R^2\le0.03)All kept (minimum R2R^2 0.94); loss 8.5× higher

The suppression thus comes from training the target, whichever regulariser prevents collapse. The protection comes from removing the gradient path, whether by a stop-gradient or a momentum target. One outcome the theory does not predict: with VICReg at ww, the constant signal (q=0q=0) is removed as well.

What generalises. In every setting we tested:

  • with a trained target, represented independent content above an innovation threshold is removed, and faster the higher its innovation;
  • without target gradients, it is kept;
  • a larger regulariser weight raises the threshold and the prediction error together.

What does not, and why.

  1. The numerical threshold. q=2λq=2\lambda is exact in the linear-Gaussian model of §2. In networks the boundary depends on the encoder (between q=0.02q=0.02 and 0.20.2), which the theory does not predict.
  2. The ordering among dependent physical variables. On Push-T, where the variables' innovations are well separated (agent below 0.001, block position below 0.01, block angle 0.18), the change in decodability follows them (rank correlation −0.93-0.93). On Cube all estimates are small and the variables are coupled, and the per-variable ordering fails. Across 177 whitened directions of the Cube image-encoder features the ordering does appear (ρ=−0.93\rho=-0.93), but there it is confounded with feature variance.
  3. The planning benefit of a stop-gradient target. Cor. 1 explains why it depends on the task: a detached target keeps content whether or not the goal depends on it, and whatever is kept enters the planning distance. In the pushing task, only stop-gradient models plan block goals better than a controller that moves the pusher and ignores the block (success 0.21 vs. 0.13; end to end: +0.00), and they do worse on pusher goals under disturbance (0.68 vs. 0.97–0.99). On Push-T a stop-gradient head does not help (44.5% vs. 46.5%).

We therefore do not propose a stop-gradient as a remedy. We claim a failure mode and diagnostics for it: measure retention and distance sensitivity next to prediction error, because which content should be kept depends on the task. We will state these boundaries explicitly in the limitations.

Thank you for the detailed responses and additional experiments. I am satisfied with the clarification and have raised my score. Note that please update the paper PDF to include the new results reported in the rebuttal.

Thank you for considering our responses and additional experiments, and for raising your score; your suggestions led to the full-scale Cube experiments and the generality tests, which have made the paper more complete. We have uploaded a revised PDF that includes all results reported in this discussion, with the changes marked in blue (mainly §4 and the new Appendix G).

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.