acceptodds
Withdrawn from ICLR 2027

You Only Live Once: Stop the Gradient or Regularize and Replay?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An agent that learns a world model from its own experience sees one finite trajectory, streamed in time order. We study JEPA in this you-only-live-once (YOLO) setting and find that the standard stop-gradient target fails silently: with finite data, the encoder can lock onto spurious noise correlations while its latent variance and training loss look healthy. An analysis of the objective motivates this failure. Removing stop-gradient and adding a variance floor regularizer prevents the collapse, and replaying past experience widens the samples for estimating the variance. Our Replay Variance Floor (RVF) recovers the latent dynamics in controlled linear systems and improves planning over the stop-gradient baseline on a range of visual control tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, world modeling, anti-collapsing, test-time training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_T47zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies JEPA representation learning from a single finite trajectory, termed the "you-only-live-once" setting. It argues that stop-gradient training can exhibit signal collapse: the encoder captures finite-sample noise correlations while latent variance, covariance, and training loss remain apparently healthy. For linear encoders and predictors, the paper derives the encoder objectives obtained after eliminating the optimal predictor. It proposes Replay Variance Floor (RVF), combining a one-sided sliced variance-floor regularizer with constant-turnover replay. Synthetic experiments evaluate held-out state recovery and rollout prediction across frequencies and trajectory lengths. Visual-control experiments use frozen DINOv2 features with a learned projector and predictor on PointMaze, Reacher, Cube, and PushT visual shifts. RVF substantially improves planning over the unregularized stop-gradient baseline in several settings, although the evidence does not establish the broad causal conclusion that stop-gradient should generally be avoided in finite-trajectory world-model learning.

Critical strengths
  1. A valuable and difficult-to-detect failure mode. The distinction between variance collapse and semantic signal collapse is important. The held-out probe and diagnostic-blindness experiment provide convincing evidence that ordinary in-sample losses and covariance diagnostics can miss the failure (Proposition 2; Figure 6).

  2. Strong synthetic mechanism studies with meaningful control gains. Tables 1–3 and 6–10 examine stop-gradient, several variance floors, replay policies, trajectory length, frequency, noise, i.i.d. sampling, EMA targets, and closed-form predictors. The maze results are also substantial: U-maze stream success improves from (21.3$\pm$11.8%) for the unregularized stop-gradient baseline to (94.9$\pm$3.1%) for RVF, and Medium-maze stream success improves from (11.8$\pm$2.3%) to (73.3$\pm$4.8%) (Table 4).

Critical weaknesses
  1. The theory does not establish the paper's central causal explanation. Theorem 1 eliminates an optimal linear predictor while holding the target fixed during differentiation, whereas the actual system jointly trains a nonlinear encoder and predictor with a moving target. More decisively, Appendix A.3 (p. 14) explicitly constructs no-stop-gradient, variance-floor solutions that encode only predictable noise while achieving zero prediction loss. Therefore the proposed objective still admits exactly the failure mode it is intended to prevent. The theorem can motivate RVF, but it cannot support the claim that removing stop-gradient makes the learned representation recover the true dynamical signal.

  2. The protocol conflates access to one finite trajectory with repeated optimization over that trajectory, and the claimed advantage is not unique to RVF. The paper's definition of YOLO permits learning from one collected trajectory, so repeated optimization is not literally inconsistent with that definition. However, Appendix B (p. 16) states that the synthetic stream wraps at its end and is optimized for 200,000 iterations; the visual experiments similarly make 10,000--50,000 updates from only 5,000 frames. The results therefore do not isolate the effects of finite data, temporal ordering, replay, and update-to-data ratio. In addition, Table 8 (p. 16) shows that at T=5000 the existing log-det floor slightly matches or exceeds RVF at every reported circle frequency under best-per-cell hyperparameter selection, while Table 4 reports a stronger Reacher-stream result for sliced+stop-gradient+CT than for RVF. Thus the evidence supports a useful finite-trajectory regularization/replay recipe, but not the broader conclusion that RVF is generally the preferred solution.

Questions for authors
  1. The paper reports the best-performing lambda separately for each cell in Tables 1–5. What metric and data split were used for this selection? Were the reported planning episodes or held-out test trajectories used at any stage, or was lambda selected using a separate validation trajectory and then evaluated on an untouched test set? Please also report the complete lambda search spaces and the constant-turnover replacement probability p.

  2. In Table 4, VICReg and SIGReg are evaluated without CT replay, whereas RVF receives CT replay and, in some rows, replay in the JEPA prediction loss. Do the visual conclusions persist when VICReg, SIGReg, and the log-det floor are evaluated with exactly the same replay buffer, replay inputs, update-to-data ratio, normalization, architecture, and hyperparameter-selection protocol? Without these matched controls, how can the gain be attributed specifically to RVF rather than to replay or additional reused prediction samples?

  3. Can the authors report an ablation over the update-to-data ratio and number of trajectory wraps, optionally including a genuinely one-pass stream as a stress test? Because the finite trajectory is repeatedly reused for optimization, this ablation is needed to distinguish the effect of RVF from that of intensive optimization over the same data.

  4. Appendix A.3 shows that the no-stop-gradient variance-floor objective still admits noise-only encoders that achieve zero prediction loss while satisfying the variance constraint. What property of RVF's optimization or replay dynamics is expected to avoid these solutions? Can the authors provide stability or basin-of-attraction evidence that explains why RVF recovers the signal despite sharing these undesirable global minima?

Reviewer_a1XsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies representation learning in JEPA under finite, single-trajectory training with sequential sampling. It focuses on “signal collapse,” a failure mode in which latent variance and training loss appear healthy, yet the representation primarily captures spurious noise correlations rather than the underlying state. The authors analyze the objectives of linear encoders and predictors to motivate this phenomenon and propose Replay Variance Floor (RVF). Experiments evaluate state recovery and multi-step prediction in synthetic dynamical systems, followed by visual control and adaptation of a pretrained world model.

Critical strengths
  • The paper investigates a meaningful failure mode that conventional diagnostics can overlook. It clearly distinguishes variance collapse from the loss of underlying signal and demonstrates that low prediction loss and healthy representation statistics do not necessarily indicate meaningful state representations. This observation offers useful insights for training and evaluating JEPA world models.

  • The synthetic experiments provide systematic controls and ablations. They cover signal frequency, trajectory length, noise level, regularization, and replay strategies, while evaluating both state decoding on independent trajectories and multi-step prediction. Controls using EMA targets and a closed-form optimal predictor help rule out some straightforward alternative explanations, strengthening the evidence for the observed failure mode.

Critical weaknesses

The methodological novelty is limited, and the existing experiments do not sufficiently establish the distinct value of the proposed modifications. RVF combines predictive learning without stop-gradient, variance-floor regularization, and experience replay. The first two components build on ideas established in prior work such as VICReg[1] and LeJEPA[2], while the replay mechanism itself is also not new. The specific methodological contribution primarily consists of applying a one-sided variance floor along random projection directions and using historical observations to estimate this statistic. However, in the visual control experiments, VICReg and SIGReg are not evaluated with the same replay configurations as RVF, making it unclear how much of the improvement comes from the proposed regularizer versus the general benefits of replay. Although the empirical findings on signal collapse are potentially valuable, the methodological contribution and its advantages over prior approaches require clearer justification.

[1] Adrien Bardes, et al. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning

[2] Randall Balestriero, et al. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics

Questions for authors
  • I wonder whether signal collapse also occurs under realistic online update budgets, given that the main experiments repeatedly train on fixed trajectories, including 200,000 updates in the synthetic setting.

  • I wonder how the observed collapse and RVF’s relative advantage change with fewer updates per observation or validation-based early stopping.

Reviewer_yU3qOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author studies world modeling from finite trajectories of agents' experience, which the author calls the YOLO (you only live once) setting. The author argues that prior work, JEPA (Joint Embedding Predictive Architecture), can fail to capture underlying dynamics in such settings and lock into spurious correlations. The author introduces Replay Variance Floor (RVF). It removes stop-gradient, applies a sliced variance floor regularizer (a modification of the prior regularizer VicReg), and uses a small replay buffer to estimate more representative latent statistics. Then, the paper provides an analysis under linear dynamics and evaluates on both synthetic and visual-control simulation benchmarks.

Critical strengths
  • The paper studies an interesting failure mode of JEPA (predictive representation learning); a non-collapsed representation can still fail to capture underlying dynamics. They also offer insight on stop-grad/non stop-grad training under finite trajectories.
  • The paper provides controlled experiments with prior regularizers, both with synthetic data and evaluation on visual control benchmarks.
Critical weaknesses

I don't fully agree with the authors' framing of the YOLO setting for agents' experience. Yes, agents live in a continuous stream of observations, but they also store and reuse their experience. For example, even if you start with a random policy in a Dreamer-like world model [1], the agent can collect some data and start learning to encode its environment reasonably well. As new data becomes available, the encoder and the world model can continue to improve. This can lead to a better policy, which collects more useful experience. Of course, such agents maintain a replay buffer, but even a relatively small buffer can help with learning the representation and dynamics. I understand that the authors also use replay, but their setup does not capture this cycle of learning and collecting new experience. I find the setup somewhat artificial and am not convinced of its practical relevance to modern model-based RL.

Even if we consider such a narrow stream of experience and force the agent to learn under these constraints, I am not sure how much I can trust the generality of the results. The theoretical analysis assumes linear encoders and predictors, which, as the authors themselves mention, does not directly extend to nonlinear settings. I am aware that the authors provide results on a few simulated visual-control benchmarks, but I don't think this is enough to suggest that the proposed technique works well in general. It's kinda interesting that the visual experiments use a frozen pretrained DINOv2 encoder (which was trained on tons of data). I think this doesn't show whether the same failure occurs when learning visual representations from scratch. VICReg and SIGReg have been studied in broader representation-learning settings, but the comparisons here are limited to the authors' setup. I think more experiments are needed, and I am still doubtful of the robustness of the technique.

For the method itself, the main idea seems to be a relatively small modification of existing regularization methods such as VICReg, along with random projections and experience replay. The variance floor is already part of VICReg, while random projections are also used in SIGReg. I understand that the authors study how these ideas work together with stop-gradient and streaming observations, but I am not convinced there is enough methodological novelty. Also, for large-scale visual representation learning, methods such as DINOv2 have shown strong results with different SSL objectives. I think the proposed method needs to be tested in more realistic representation-learning settings to better understand its usefulness.

[1] Hafner, Danijar, et al. "Mastering diverse domains through world models." arXiv preprint arXiv:2301.04104 (2023).

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.