acceptodds
Under review as a conference paper at ICLR 2027

When Anti-Collapse Objectives Have Collapsed Optima

Abstract

Every JEPA carries collapse-prevention machinery, and a recent line of work replaces the older heuristics with a single distribution-matching objective on the latent (LeJEPA; Rectified LpJEPA). We ask when such an objective is safe. An objective that reads the latent only through per-feature scalar statistics has a collapsed global optimum when its entrywise function is convex and non-increasing (softplus, hinge); for the minimisers are single-entry, rank-one latents. A softplus non-negativity penalty — the form a rectified target takes when written as a penalty — placed at the encoder of tabular T-JEPA degrades runs under its standard linear probe, to the majority-class share, locked within five epochs. The collapse is geometric rather than informational: the re-examined latents sit on one common vector with a residual of of its scale that still carries the label — a standardised probe recovers most of it, yet stored in bf16 or fp16 every input rounds to the same vector. Under a fair probe the placement effect survives but shrinks: pp for this penalty (from pp), –pp for everyday and hinge penalties (from pp), and no encoder-side deficit for the published Rectified-LpJEPA loss. A variance-floor repair restores information but not scale, and along the collapsed direction its restoring gradient vanishes with the residual while the published sliced-Wasserstein loss keeps it. A one-forward-pass check that scores objectives at a collapsed and a healthy latent flags every penalty that collapses deterministically, but it is a warning sign, not a certificate: in pre-registered runs the reference LeJEPA objective, which it scores as safe, collapses T-JEPA to the majority share when placed at the predictor ().

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, representational collapse, self-supervised learning, regularization, distribution matching, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kHsiOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to understand in which conditions would a JEPA model collapse during training (i.e., returns a trivial mapping of latent state representation). The paper tries to study different penalty/regularisation for the loss for T-JEPA models, and also at different levels of training precision, finding different amounts of collapse during training. It also provides some simple fix to the issue (like using a variance floor).

Critical strengths
  • There are some attempts to tackle a problem which can potentially improve the understanding of JEPA models. This would be useful as the JEPA models gain more popularity and usage in practice.
Critical weaknesses
  • The writing is hard to follow in general, and many parts require several re-reads to understand. Some of my examples for this are:

    • In the first two sections, I feel it is not so immediately clear to follow with what the work is exploring or doing (especially if the reader is already not familiar with the area). The experimental setup for the paper remains unclear even after reading this point.
    • In the proceeding sections, the results are not organised in a manner that can be easily/logically followed, but rather as scattered findings.
    • Much of the results seem to be relegated to the appendix which may affect the flow of the writing, since I would have to check with the appendix to see the results that support the statement made in the paper. It is likely that not all the parts in the paper is required (for example, the very specific details about which seeds give what characteristics), and that some of the details can be placed in the appendix in favour of some more representative results.
    • While there are some attempts to summarise the results, it still remains unclear from reading as to what is the main outcome of the work, and in which context it may be useful. I feel this would be useful to state earlier on, and with some empirical demonstration that can be useful in a wider context for JEPA.

    I believe major editing to the writing would be required to make the paper more presentable and understandable (potentially beyond what would be possible during the rebuttal period).

  • While JEPA is a useful model to study, the paper seems to make little attempt to frame their finding as to why it is useful in practice. I feel there are limited discussion (e.g., in Section 7) as to implications of the work to practical JEPA training, but this discussion is limited. It would benefit from a more practical discussed use cases or from a better set of experiments/demonstrations to how the finding can be used. As of now, the findings seem unclear as to what it is, and how it can be useful in practice. This point also somewhat links back to the comment about the writing from the above, that the paper seems to make it remain unclear.

  • For the results specifically: experiments are presented in a way that seems doubtful to its generalisability (t seems to fixate itself on specific seeds of the training, and is unclear as to whether the findings are true outside the settings provided in the paper), while the theoretical results are unclear as how closely related they would be to the practical T-JEPA training scenarios for them to be applicable.

Other minor typos (not consequential to my decision)

  • Line 98: indicator variable \mathbb{1} rendered incorrectly (I think need to use \mathbbm{1} or something)
Questions for authors
  1. What is the motivation behind the work provided, and how can the findings of the paper be useful for people using or training T-JEPA models in practice? What proposed changes might be possible to the conventional T-JEPA training currently?

  2. Why does the paper study T-JEPA models specifically? Would the findings be generalisable to other variants of JEPA models as well?

Reviewer_1kNSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when a JEPA collapse-prevention objective can itself favor a collapsed representation. Its theoretical result concerns entrywise latent penalties under a fixed second-moment constraint: convex non-increasing penalties such as softplus have a constant collapsed global optimum, while the ℓ1\ell_1 case has a single-entry rank-one minimizer. The main experiment places a softplus non-negativity surrogate, called RECTPEN, either on the encoder output or on the predictor output in tabular T-JEPA. All 28 encoder-placement runs degrade under the standard probe and 24 reach the majority-class share; a fresh paired grid gives 15/15 strict collapses at the encoder and 0/15 at the predictor.

The authors then examine what the word “collapse” means in this setting. Frozen latents are almost constant relative to their mean, yet a residual of around 2×10−72\times10^{-7} of the mean scale still carries label information: standardized linear probes recover 0.634–0.701 accuracy on selected checkpoints where the raw probe often returns 0.460. Storing those latents in bf16 or fp16 before centering destroys the residual. Controls with the actual Rectified LpJEPA distribution-matching loss show 0/9 encoder collapses at both 100 and 300 epochs, while a reference LEJEPA objective can fail under predictor placement. The paper also tests ℓ1\ell_1 and hinge penalties, a variance-floor repair and a one-pass collapsed-versus-healthy objective check. Its own evidence shows that this check can warn of some deterministic failures but cannot certify safety.

Critical strengths

The most convincing part is the encoder/predictor placement experiment. The authors hold data, seed and source configuration fixed and change the tensor to which RECTPEN is applied. The fresh grid yields 15/15 strict failures at the encoder and 0/15 at the predictor; the result also survives a wide sweep of regularization weights. This supports a concrete implementation warning: the same apparent regularizer can have quite different effects depending on where its gradient enters the online network. The measured eight-order drop in the prediction gradient reaching the encoder during the failure window is a useful, though not conclusive, mechanistic clue.

The paper is unusually careful about controls that could otherwise make the result misleading. It names RECTPEN as a surrogate rather than equating it with the published RDMREG objective, and tests the published loss separately. The ℓ1\ell_1 and hinge follow-ups show that the failure is not confined to one hand-built softplus penalty: both collapse at encoder placement in 18/18 runs across the tested data/seed cells and neither does so at predictor placement. The fair-probe and half-precision checks also show why majority-class probe accuracy alone is an incomplete diagnosis. Finally, the one-pass score fails on some objectives that collapse in training, a limitation the paper quantifies rather than hiding.

Critical weaknesses
  • What the theorem predicts in practice remains limited. It characterizes global optima under a normalization constraint, whereas the training experiments involve finite batches, a prediction loss, adaptive optimization and a nonconvex encoder. The gap between “a collapsed optimum exists” and “training collapses” is illustrated by the paper's own counterexamples: the one-pass check marks some objectives safe even though they collapse under a particular placement, and the same entrywise penalties do not produce strict collapse in the small vision and time-series boundary studies. The paper acknowledges this, but the broad phrasing about objective classes may still lead a reader to overgeneralize from a convincing T-JEPA result.

  • The named objective controls are not fully comparable. The main SIGREG grid uses a standardized Cramér–von Mises variant with 32 fixed directions, whereas the LEJEPA reference uses Epps–Pulley with 256 resampled directions. The published Rectified LpJEPA control reproduces the loss but initially not the full architecture around ReLU, though the authors do add a ReLU ablation. The striking 6/9 predictor-placement failures for reference LEJEPA are reported without retuning the regularization coefficient despite a change in token count. Those are worthwhile observations, but they do not yet establish a general safety ordering among published methods. I would keep the strongest claim focused on the objective/placement pairs actually tested.

Questions for authors

Can the authors report the effective per-token loss and gradient scales for the reference LEJEPA objective at the encoder and predictor and rerun that 6/9 predictor failure after normalizing for token count or retuning λ\lambda? This would determine whether the apparent reverse placement effect is intrinsic to that objective or a scale artifact.

For the theoretical result, could the authors spell out how closely the unit second-moment constraint used in Proposition 1 matches the scale control in the implemented RECTPEN objective during the first few epochs, when collapse becomes irreversible? The fixed-scale theorem and the observed optimizer trajectory are both interesting; the connection between them is important to the causal claim.

Reviewer_vcFDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors argue that some regularizers added to JEPA models to stop collapse can end up causing it instead. They further show that “entrywise” objectives, such as non-negativity and L1 sparsity penalties, have a collapsed optimum, and show this in tabular T-JEPA. Putting such a penalty on the encoder output makes the model collapse almost every time, while putting it on the predictor output works fine. They also suggest a quick check to spot risky losses before training, though it doesn’t catch every case.

Critical strengths

I think the idea has some merit i.e some regularizers meant to prevent collapse can actually favor a collapsed representation, and where the regularizer is applied (encoder vs predictor) matters. But despite this, in my assessment this does not outweigh the weaknesses as listed below.

Critical weaknesses
  1. The abstract, along with the complete paper, is really hard to make sense of. The sentences are unnecessarily convoluted and hard to follow. There are too many numbers packed into every sentence, which makes it hard to read. Many sentences use “;” (about a third of the sentences in the main text), which is not normal. An example is the Conclusion paragraph, which is very hard to follow (e.g., “A guarantee must name the tensor and the collapse; a re-implementation, the mechanism.”).

  2. The main text mentions data files and scripts such as “data/collapsed_probe_analysis.csv” and “src/verify_paper_numbers.py”. This is not formal paper writing. Such details belong in the reproducibility statement or the appendix, not in the main text. I also spot a lot of notation issues.

  3. The naming is quite confusing. There are RECTPEN, RDMReg, a code flag called “rdmreg” that actually refers to RECTPEN, SIGReg-CvM and reference SIGReg, and I kept mixing them up while reading. The same goes for collapse: the paper uses strict, soft, strong, majority-exact and degraded, which seem to overlap, and it is hard to keep track of which one is meant where.

  4. There is language like footnote 1: “Untelemetried runs count as predictor-era, so five healthy post-fix runs may sit in that stratum — a misplacement that can only add healthy runs to that stratum, never remove a collapse from it.” It is unclear what this means, and it reads like internal notes rather than a paper.

Overall, I found it very difficult to understand what the authors are trying to convey.

Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.