- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies JEPA representation learning from a single finite trajectory, termed the "you-only-live-once" setting. It argues that stop-gradient training can exhibit signal collapse: the encoder captures finite-sample noise correlations while latent variance, covariance, and training loss remain apparently healthy. For linear encoders and predictors, the paper derives the encoder objectives obtained after eliminating the optimal predictor. It proposes Replay Variance Floor (RVF), combining a one-sided sliced variance-floor regularizer with constant-turnover replay. Synthetic experiments evaluate held-out state recovery and rollout prediction across frequencies and trajectory lengths. Visual-control experiments use frozen DINOv2 features with a learned projector and predictor on PointMaze, Reacher, Cube, and PushT visual shifts. RVF substantially improves planning over the unregularized stop-gradient baseline in several settings, although the evidence does not establish the broad causal conclusion that stop-gradient should generally be avoided in finite-trajectory world-model learning.
Critical strengths
-
A valuable and difficult-to-detect failure mode. The distinction between variance collapse and semantic signal collapse is important. The held-out probe and diagnostic-blindness experiment provide convincing evidence that ordinary in-sample losses and covariance diagnostics can miss the failure (Proposition 2; Figure 6).
-
Strong synthetic mechanism studies with meaningful control gains. Tables 1–3 and 6–10 examine stop-gradient, several variance floors, replay policies, trajectory length, frequency, noise, i.i.d. sampling, EMA targets, and closed-form predictors. The maze results are also substantial: U-maze stream success improves from (21.3$\pm$11.8%) for the unregularized stop-gradient baseline to (94.9$\pm$3.1%) for RVF, and Medium-maze stream success improves from (11.8$\pm$2.3%) to (73.3$\pm$4.8%) (Table 4).
Critical weaknesses
-
The theory does not establish the paper's central causal explanation. Theorem 1 eliminates an optimal linear predictor while holding the target fixed during differentiation, whereas the actual system jointly trains a nonlinear encoder and predictor with a moving target. More decisively, Appendix A.3 (p. 14) explicitly constructs no-stop-gradient, variance-floor solutions that encode only predictable noise while achieving zero prediction loss. Therefore the proposed objective still admits exactly the failure mode it is intended to prevent. The theorem can motivate RVF, but it cannot support the claim that removing stop-gradient makes the learned representation recover the true dynamical signal.
-
The protocol conflates access to one finite trajectory with repeated optimization over that trajectory, and the claimed advantage is not unique to RVF. The paper's definition of YOLO permits learning from one collected trajectory, so repeated optimization is not literally inconsistent with that definition. However, Appendix B (p. 16) states that the synthetic stream wraps at its end and is optimized for 200,000 iterations; the visual experiments similarly make 10,000--50,000 updates from only 5,000 frames. The results therefore do not isolate the effects of finite data, temporal ordering, replay, and update-to-data ratio. In addition, Table 8 (p. 16) shows that at T=5000 the existing log-det floor slightly matches or exceeds RVF at every reported circle frequency under best-per-cell hyperparameter selection, while Table 4 reports a stronger Reacher-stream result for sliced+stop-gradient+CT than for RVF. Thus the evidence supports a useful finite-trajectory regularization/replay recipe, but not the broader conclusion that RVF is generally the preferred solution.
Questions for authors
-
The paper reports the best-performing lambda separately for each cell in Tables 1–5. What metric and data split were used for this selection? Were the reported planning episodes or held-out test trajectories used at any stage, or was lambda selected using a separate validation trajectory and then evaluated on an untouched test set? Please also report the complete lambda search spaces and the constant-turnover replacement probability p.
-
In Table 4, VICReg and SIGReg are evaluated without CT replay, whereas RVF receives CT replay and, in some rows, replay in the JEPA prediction loss. Do the visual conclusions persist when VICReg, SIGReg, and the log-det floor are evaluated with exactly the same replay buffer, replay inputs, update-to-data ratio, normalization, architecture, and hyperparameter-selection protocol? Without these matched controls, how can the gain be attributed specifically to RVF rather than to replay or additional reused prediction samples?
-
Can the authors report an ablation over the update-to-data ratio and number of trajectory wraps, optionally including a genuinely one-pass stream as a stress test? Because the finite trajectory is repeatedly reused for optimization, this ablation is needed to distinguish the effect of RVF from that of intensive optimization over the same data.
-
Appendix A.3 shows that the no-stop-gradient variance-floor objective still admits noise-only encoders that achieve zero prediction loss while satisfying the variance constraint. What property of RVF's optimization or replay dynamics is expected to avoid these solutions? Can the authors provide stability or basin-of-attraction evidence that explains why RVF recovers the signal despite sharing these undesirable global minima?