acceptodds
Under review as a conference paper at ICLR 2027

The JEPA Paradox in Language: The Geometry of Linguistic Alternatives

Abstract

Joint-Embedding Predictive Architectures (JEPAs) are effective for images, video, and audio, yet deterministic JEPA-style latent prediction has not become a standard objective for text encoders. We argue that this gap reflects a mismatch between squared-error latent prediction and the conditional structure of language. The key requirement is conditional concentration: given a context and target location, the target representation should lie near a single meaningful point. Local image prediction often satisfies this through spatial continuity, whereas masked text can admit multiple valid token or span completions whose representations need not share a coherent center. We formalize this mismatch through three conditions — predictability, non-collapse, and low conditional variance — and show how their failure creates centroid degeneracy and collapse pressure in text. Matched I-JEPA and T-JEPA experiments reveal the predicted sequence: mutual-information saturation and elevated target variance precede train–validation instability, effective-rank degeneration, cosine collapse, and poor downstream transfer. The same pattern appears across five independent data seeds, indicating that it is not a sampling artifact. These results do not rule out predictive learning for language; they show that text-compatible JEPA objectives must preserve multiple plausible completions rather than compress them into a single latent point.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-Embedding Predictive Architecture (JEPA), Self-Supervised Learning, Language Representation Learning, Predictive Representation Learning, Representation Collapse, Conditional Uncertainty, Latent Prediction, Masked Language Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_juYCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents theoretical analysis and empirical studies about the JEPA for text (T-JEPA). The motivation is that "although JEPA have been widely adopted in other modalities, its application to text is under-explored".

The authors tried to explain their contributions in Abstract and Introduction. However, if we take a look at Table 1, we see something very strange

  1. Except BERT, all baselines have an origin in Computer Vision (CV) domain, rather than in Natural Language Processing (NLP). This makes the evaluation very biased, and cannot reflect the state-of-the-art in NLP domain.
  2. BERT outperforms T-JEPA by an extremely huge margin, indicating that the text-JEPA is never working in this paper
  3. T-JEPA is either comparable to, or worse than other baselines, such as BYOL or VICReg Details of the above weaknesses will be discussed in the Weakness session.

I guess the authors may say "finding methods that do not work is also a contribution". However, I think this is only meaningful when the method is believed to be strong and we find some bugs of this strong method. Currently, all methods are apparently weaker than BERT, I don't think there is sufficient scientific insight.

Given the missing context in NLP domain, extremely weak performance in downstream tasks, as well as the fact that BERT can be interepreted as the discrete version of JEPA, I recommend a "clear reject" to this submission.

Critical strengths

None

Critical weaknesses

W1: BERT can be considered as the discrete version JEPA

Since JEPA is an evolving paradigm, I do not want to define this paradigm with strict criteria. However, I believe a general interpretation of JEPA is

  • An latent encoder encodes raw input into a latent space. The procedure is often coupled with various regularizations to prevent latent mode collapse (e.g., the encoder maps all signals to the same embedding).
  • Another module conducts mask reconstruction inside that regularized latent space.

Then let us take a look at the BERT

  • A tokenizer encodes text signals into a set of finite indices, which we call vocabulary or codebook.
  • Each index in the vocabulary is assigned with a high-dimentional embedding or vector. These embeddings are often randomly initialzed, and will be updated during the training.
  • a Transformer will conduct mask reconstruction for the aforementioned finite set of embeddings

If we consider the discrete tokenization and the update of finite embeddings as a kind of regularization, then BERT is also conducting mask reconstruction task in a regularized space. Consequently, BERT can be considered as the discrete version of JEPA.

In this context, the claim that "JEPA is under-explored for text" itself becomes invalid.

W2: The paper is mostly written from a Computer Vision context, lacking up-to-date NLP contexts

BYOL or VICReg, to my knowledge, have an origin in CV domain.

The only NLP-derived encoder in this paper is BERT, completely ignoring other BERT variants, GPT-style variants, T5-style variants, as well as the CLIP-style text encoders.

I respect CV as a research field. Nevertheless, the ignoring of all these NLP baselines is inacceptable for this paper.

W3: Weak performance

The Table 1 shows that BERT largely outperform all other baselines. Moreover, other baselines can outperform T-JEPA in some datasets and some metrics.

Although the authors claim that they found the problems of T-JEPA, the weak performance suggests that the T-JEPA model is still not well trained, and there are more severe problems need to be addressed.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Why BERT Differs from JEPA, Why Our Baselines Are Appropriate, and Why T-JEPA Is Not Under-Trained

We appreciate the reviewer's perspective. However, we would like to clarify an important distinction between the two objectives.

[Response to W1]

In BERT, the prediction target is a discrete token from a finite vocabulary, and the model predicts a probability distribution over the vocabulary. In contrast, JEPA predicts a continuous target representation using a deterministic predictor trained with a squared-error objective (Eq. 5). Consequently, the optimal JEPA predictor is the conditional expectation of the target representation, which may not correspond to any valid target when multiple plausible continuations have substantially different representations. Therefore, although BERT can be viewed as masked prediction in a learned discrete representation space, its distributional prediction objective differs from the deterministic latent regression studied in our work.

Furthermore, Baevski et al. (2022) explicitly distinguish data2vec's contextualized latent targets from BERT's original token-level targets. Assran et al. (2023) also distinguish masked-token reconstruction methods from JEPA in their discussion of self-supervised learning paradigms.

Importantly, even under the reviewer's broader interpretation of JEPA, our central conclusion remains unchanged: distributional prediction over discrete targets can preserve multiple plausible alternatives, whereas deterministic regression to a single continuous target may suffer from the conditional-variance issue identified in our analysis. We also clarify that our claim is narrower than stating that JEPA is under-explored for text in general. Rather, we investigate the limitations of deterministic, squared-error latent prediction as a standalone objective for text encoders.

[Response to W2]

BYOL and VICReg indeed originate from the Computer Vision domain. However, our research question is why the JEPA paradigm, which originated in vision, cannot be directly transferred to text encoders, in the sense of BERT-style models, as it is in the original I-JEPA. These methods therefore serve as relevant non-contrastive self-supervised learning baselines, alongside BERT as a representative NLP pretraining objective.

Regarding GPT-style and T5-style models, our study deliberately focuses on how JEPA can be applied in the encoder-only scenario, to maintain a setting comparable to the original I-JEPA formulation. Moreover, causal language modeling and sequence denoising objectives predict distributions over discrete tokens or token sequences, which differ from the deterministic latent regression objective investigated in our work. We refer the reviewer to Appendix J for further motivation of our choice of baselines.

Regarding CLIP-style text encoders, CLIP is trained on paired image-text data with cross-modal contrastive supervision, i.e., it learns multimodal embeddings. This differs from our focus on single-modality self-supervised learning for text encoders, where the main question is why JEPA itself cannot be applied to text as it is to images.

We clarify that we did not ignore NLP baselines. As discussed in Appendix J, we consider LLM-JEPA and DLLM-JEPA, as well as BERT-JEPA, which combines the MLM loss with a JEPA loss. These approaches incorporate the JEPA objective only as an auxiliary loss alongside their primary objectives, such as causal language modeling, diffusion, or MLM. Removing these primary objectives leads to collapse or a substantial degradation in performance, so the JEPA objective does not stand alone in these works. This observation is consistent with our investigation of JEPA as a standalone objective, rather than as an auxiliary component of generative pretraining.

[Response to W3]

We would like to emphasize that our work does not aim to propose a new method or outperform existing NLP pretraining approaches. Instead, this is a diagnostic study that investigates the failure of deterministic JEPA when applied to text, in contrast to its successful application in vision. Our goal is to provide a theoretical and empirical understanding of this failure, which can serve as a foundation for future research on JEPA for text.

Regarding the concern that T-JEPA may simply be under-trained, our analysis identifies a specific theoretical limitation of the objective itself. In particular, we observe that the conditional structure of masked text prediction leads to elevated conditional variance, which is associated with the subsequent degeneration of learned representations. Therefore, the poor downstream performance is not merely an isolated empirical observation, but is consistent with the failure mechanism investigated in our work.

As mentioned, our work is a diagnostic study, where T-JEPA’s failure is the phenomenon we explain, not a method we propose. To rule out under-training, we rely on several controls. I-JEPA trains successfully under the same recipe, and BERT performs strongly with the same backbone, tokenizer and data, so the pipeline is sound. The failure follows the theory’s predicted order (MI saturation and elevated conditional variance before rank and cosine collapse), and this order holds across five seeds. BYOL with identical tuning collapses under masking but stays competitive under replacement, which points to conditional ambiguity rather than optimization as the cause. Shorter warmups and fixed learning rates make T-JEPA collapse earlier, not later. Finally, its chance-level accuracy with single-class F1 indicates a near-constant representation (cosine near 1), not a partially trained encoder.

Dear authors, I appreciate your polite and patient explanations. I respect your faith in this research topic.

Nevertheless, I do not think these responses address the critical concerns.

To ACs, I still don't think it is valuable to present a failure collection of CV-inspired yet NLP-targeted text representation learning methods in ICLR2027. Such failure collection can be presented in a blog, in a shorter paper, or in a workshop.

I will keep my original rating.

Reviewer_znQBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why JEPA-style latent prediction works well on vision/audio, but fails on text encoders. The main argument is that L2 regression requires the target representation to concentrate around a single point given the context. Image patches usually satisfy this, however, masked text tokens often have multiple reasonable completions, so the predictor just regresses to their centroid. The authors formalize this into three conditions (predictability, non-collapse, and low conditional variance), showing that violating them leads text JEPA to collapse.

For experiments, the authors train I-JEPA on 100K ImageNet and T-JEPA on 100K C4 sentences using the exact same optimization schedule (with 5 seeds). Results show T-JEPA has low MI, unstable training, and finally collapses in both effective rank and cosine similarity. Its frozen features only get chance-level performance on IMDB/SNLI and 0 score on retrieval.

Critical strengths
  1. Good research question with testable answer. Conditional concentration gives a concrete and falsifiable explanation why latent MSE prediction works better on images than text. It also suggests some clear remedies like mixture predictors or semantic-level targets. The discussion in App. K is also nice: marginal regularizer like SIGReg only controls total covariance, so it cannot guarantee targets are predictable from context. If this hypothesis holds, it provides a useful angle for designing text JEPA in the future.

  2. Transparent diagnostic setup. The paper reports several complementary metrics on both splits, repeats text runs over 5 data seeds, and explains clearly why the schedule is chosen. This setup is very helpful for doing follow-up ablations.

Critical weaknesses
  1. The cause of failure is not clearly separated from simple optimization collapse. App. D mentions that effective rank is near-degenerate from the first epochs, cosine similarity stays near 1.0 (actually already ~1.0 at step 0 in Fig. 9), and MI stays near 0. Since all three metrics are bad from the beginning, we cannot see the claimed order that predictability fails first and then collapse happens later. This pattern just looks like the usual early collapse in non-contrastive methods when there is no target normalization or variance term. In contrast, VICReg and Barlow Twins trained with same masking do not collapse at all (Table 2).

  2. It is not convincing that the problem is specific to text. The propositions in App. B actually apply to both image and text, the difference between modalities only comes from the assumptions made by authors. Also, the variance for the two modalities is estimated in totally different ways, so the two numbers are not directly comparable. More importantly, data2vec (cited in the paper) also uses deterministic latent regression on masked text with normalized targets, and it can match RoBERTa on GLUE (82.7 vs. 82.5). Without explaining why data2vec works, the failure observed here may just come from this specific T-JEPA training recipe rather than language itself.

Questions for authors
  1. How are effective rank, cosine similarity and MI computed (pooled or per token, train or eval mode)? Why is the cosine about 1 at step 0, and why is the I-JEPA rank so low?
  2. Are the target outputs normalized, as in I-JEPA and data2vec? What is the batch size, and does the instability at step 9,340 change if the learning-rate schedule is changed?
  3. Why do all non-MLM baselines score exactly 0 on FEVER?
  4. How do you explain the success of data2vec on text?
2 comments
AuthorsOpenReview ↗

Target Normalization and SIGReg Do Not Prevent T-JEPA Collapse; data2vec Consistent with Our Analysis

We thank the reviewer for an exceptionally careful review. The reviewer is right on several points, and we report what we found.

Q1. Effective rank and cosine are computed per token over all 256 positions, including padding. Train-mode records include BERT dropout (p=0.1), whereas our ViT uses none, so train-mode text metrics measure dropout noise and are not comparable to I-JEPA. Cosine is uncentered. The step-0 cosine ~1 reflects rank collapse at initialization in a randomly initialized post-LN Transformer (Dong et al., 2021; Noci et al., 2022) combined with the uncentered metric; it is not evidence of learned collapse. About MI proxy, it paired the full-sentence context encoding with the full-sentence target (not context vs. masked span), and the positive was enqueued among the negatives.

Q2. Targets are not normalized in either model beyond the encoder’s affine final LayerNorm, so the target scale can shrink. The training loss falling by ~8 orders of magnitude is consistent with this shortcut. Batch size is 64 with peak LR 1e-3 and 67% warmup. Step 9,340 came from our main unseeded run; across the five seeded runs, loss spikes occur at seed-dependent steps clustering near the LR peak, consistent with the reviewer’s optimization concern.

Following the reviewer’s suggestions, we ran the following additional experiment:

MethodIMDBSNLISQuAD F1FEVER nDCG@10MS MARCO nDCG@10Eval-mode rank
BERT (MLM)81.660.616.10.1550.095441
data2vec70.551.012.00.0250.043336
VICReg58.042.78.60.0000.000285
T-JEPA50.032.82.2001.0
T-JEPA + normalized targets50.032.82.2001.0
T-JEPA + SIGReg50.032.82.2001.0

All three T-JEPA variants collapse to an input-independent output: the probe can only predict the majority class. Target normalization does not prevent this. With SIGReg, the regularizer is satisfied by dropout noise rather than input information: rank is 509 with dropout on and 1.0 in eval mode, and about 50% of the variance comes from noise. This is consistent with App. K, where a marginal constraint does not enforce input dependence. data2vec, however, learns useful features, so deterministic latent regression on text is not inherently doomed.

Q3. For data2vec, a non-MLM model evaluated with the same pipeline, scored 0.025 nDCG@10. The MTEB FEVER corpus contains about 5.4M documents, so an encoder whose embeddings carry little input information has a near-zero chance of placing a relevant document in the top 10. Random ranking gives an expected Recall@100 of about 2×10⁻⁵. For T-JEPA, whose output is constant across inputs, all documents tie and the score is exactly 0. VICReg’s 0.000 is a small positive value rounded to three decimals.

Q4. data2vec is consistent with the analysis. Its Smooth-L1 (β=4) is effectively quadratic on LayerNorm targets, so Eq. 58 applies.

(i) LN fixes ∣n(v)∣2=d\\|n(v)\\|^2 = d, so Prop. B.16 becomes

Var(n∣xC)=d∑u,vpupv(1−cos⁡(n(u),n(v))),\mathrm{Var}(n \mid x_C) = d \sum_{u,v} p_u p_v \bigl(1 - \cos(n(u), n(v))\bigr),

which means variance can shrink only by merging directions, not by shrinking norms.

(ii) Write nℓ(v)=sℓ(xC)+rℓ(v;xC)n_\ell(v) = s_\ell(x_C) + r_\ell(v; x_C) (shared context + token residual), with cross-layer correlations κs,κr\kappa_s, \kappa_r. The relative conditional variance of the top-KK average falls below the single-layer value iff κr<κs\kappa_r < \kappa_s, by up to K×K\times (K=10K=10), shrinking Δ\Delta in Prop. B.17. This matches data2vec's ablation, where multi-layer targets beat K=1K=1.

Dear Reviewer znQB,

Thank you again for your careful review. Following your suggestions, we ran additional experiments and clarified the key points in our response:

  • Optimization vs. conditional ambiguity: Adding target normalization or SIGReg does not prevent T-JEPA collapse (eval-mode rank 1.0 in both cases). With SIGReg, the regularizer is satisfied by dropout noise rather than input information, which is consistent with App. K.
  • Metrics: The step-0 cosine ~1 comes from rank collapse at initialization combined with an uncentered metric. We now state clearly that it is not evidence of learned collapse.
  • data2vec: It learns useful features (IMDB 70.5, SNLI 51.0) and is consistent with our analysis. LayerNorm targets with a Smooth-L1 loss that is effectively quadratic, and multi-layer averaging, reduce the relative conditional variance.
  • FEVER scores: We explain why the near-zero values are expected for the non-MLM baselines.

We hope these additions address your concerns about separating our mechanism from simple optimization collapse and about data2vec. We would appreciate it if you could let us know whether they resolve your concerns, and we would be happy to answer any follow-up questions before the discussion closes.

Thank you for your time.

Authors

Reviewer_e3oGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper essentially argues that JEPA losses are not well suited for text as unlike images blanked out pieces of text can diverge meaningfully much more. Therefore the centroid-like losses used in vanilla JEPA do not fit as the valid continuations can be much more spread out in representation space. They make an alternative loss to fix this and do some empirical studies (which do not seem very convincing)

Critical strengths

The core idea is interesting. Indeed, text and images often differ critically in terms of the modal image and the modal piece of text, and in case of the latter, a centroid loss is indeed going to collapse choices into a centroid that has not much value by itself. I am mostly swayed by the paper's core observation for the USUAL case in each modality, but I do not think enough analysis was done for the edge elements (see below)

Critical weaknesses

A very easy way to test the hypothesis would be images of written text. Here, if the authors were correct, the classical JEPA loss, when working over blanked out sections of an image of a textbook page, would show massive errors, especially if the candidate words appeared meaningfully different when written down.

Secondly, the empirical studies are not convincing (BERT beats all models tested). The authors say that this is anticipated, but do not outline a clear use case for this alternative loss for practitioners clearly that I could see.

Thirdly, very little seems to have been devoted to seeing if modern LLM representation spaces conform to the hypothesis of "massively branching options" even though enough models are open source to extract embedding choices.

Fourth, little to no analysis of K-modal loss functions (if indeed most blanked out words can be thought of as choices centered around <=K key words) such as a K-component GMM-induced loss.

Questions for authors

See weaknesses (Q1-4)

3 comments
AuthorsOpenReview ↗

Our Paper Is a Diagnostic Study, Not a New Method

Thank you for the review. We have one clarification: the paper does not propose an alternative loss. It is a diagnostic study, and T-JEPA is a controlled probe (see our general comment).

(Q1) Our theory concerns the conditional structure of targets rather than pixels vs. tokens; it therefore predicts that JEPA on rendered text should degrade when whole words are masked but not when masks fall within a character. Importantly, our claim is not that images inherently admit concentrated targets while text inherently does not. Rather, conditional concentration is a property of the context–target relationship, independent of the input modality. Therefore, rendered text is not a counterexample to our theory: masking a complete word can introduce multiple semantically valid target alternatives, even when the input is represented as pixels. Conversely, masking a sufficiently constrained character fragment may yield a concentrated target distribution. This distinction follows directly from our formulation in Section 4 and reinforces that conditional geometry, rather than modality identity, is the relevant criterion. We view the proposed experiment as a valuable extension, but it is not necessary to establish the mechanism studied in this work.

(Q2) The takeaway is a design principle, as text JEPAs must preserve multiple plausible completions with distributional, mixture-based or semantic-level targets. We respectfully clarify that the reviewer evaluates our work against a methodological contribution that we do not claim to make. This paper does not propose a competing text-pretraining objective; it investigates why deterministic point-prediction objectives can fail under multimodal conditional targets. Accordingly, BERT outperforming T-JEPA is consistent with, rather than contradictory to, our central hypothesis: distributional prediction preserves uncertainty over alternative token completions, whereas deterministic squared-error regression reduces them to a single conditional mean.

More importantly, our evidence does not rely on downstream performance alone. Sections 5.2–5.4 establish a temporal failure signature through complementary diagnostics: information saturation and elevated conditional variance precede optimization instability and representation degeneration. The downstream experiments in Section 5.5 then examine the consequences of this failure for representation transfer.

The practical implication is an objective-selection principle: deterministic latent prediction is appropriate when conditional targets are sufficiently concentrated; when they are multimodal, the objective must retain their distributional structure rather than force a single-point prediction. Developing a particular alternative objective is a separate methodological direction, not the contribution claimed here.

(Q3) We only measure this in pretrained text encoders (e.g., BERT/RoBERTa), consistent with our encoder-only scope. Our empirical scope is intentionally restricted to encoder-only models and the direct application of deterministic latent prediction to masked textual targets. Extending the analysis to autoregressive LLMs would introduce a different prediction setting, with causal conditioning and generative pretraining objectives that are absent from our controlled setup. We nevertheless discuss recent LLM-JEPA and DLLM-JEPA approaches in Appendix J, highlighting that their JEPA objectives remain coupled with distributional generative objectives. Thus, our findings should not be interpreted as a claim about all modern LLM representation spaces; they concern the standalone deterministic masked-target prediction setting studied in this work.

(Q4) Mixture-based predictors are exactly the remedy our analysis motivates (Sec. 1, App. B). We agree that mixture-based prediction is a natural direction suggested by our analysis. However, evaluating a K-component GMM predictor would introduce additional modeling choices, including the number of components, mixture-weight estimation, and mode assignment, which constitute a separate methodological investigation. Our objective here is to establish the underlying limitation of deterministic point prediction, not to benchmark alternative probabilistic architectures. We explicitly identify mixture-based and distributional prediction as potential directions in Section 1. Importantly, our conclusions do not depend on a particular choice of alternative predictor: they establish why collapsing multiple plausible targets into a single conditional mean can be problematic in the first place.

We thank the reviewer and first clarify a misunderstanding: we do not propose a new loss. Our paper is a diagnostic study explaining why standard deterministic JEPA fails on text (App. M), so BERT outperforming T-JEPA is the expected outcome under our analysis, not a weakness of a proposed method. The takeaway is a design principle, as text JEPAs must preserve multiple plausible completions with distributional, mixture-based or semantic-level targets. We find the reviewer’s suggested tests valuable and complementary. For (Q1), our theory concerns the conditional structure of targets rather than pixels vs tokens. It therefore predicts that JEPA on rendered text should degrade when whole words are masked but not when masks fall within a character. For (Q3) we only measure this in pretrained text encoders (e.g., BERT/RoBERTa), consistent with our encoder-only scope. (Q4) Mixture-based predictors are the remedy our analysis motivates (Sec. 1, App. B).

Dear Reviewer e3oG,

Thank you again for your review. As the discussion period is closing, we would like to briefly summarize how our response addresses your four concerns:

  • Scope: The paper is a diagnostic study, not a new method. T-JEPA is a controlled probe, so BERT outperforming it is the expected outcome under our analysis.
  • Q1 (rendered text): Our theory concerns the conditional structure of targets, not the modality. It predicts degradation when whole words are masked and not when masks fall within a character. We agree this is a valuable extension.
  • Q3 and Q4: We scope the study to encoder-only models and discuss LLM-JEPA variants in App. J. Mixture-based predictors are the remedy our analysis motivates, and we leave their evaluation to future work.

We would be grateful to know whether this resolves your concerns, in particular the Soundness score of 1, which seems to stem from reading the paper as proposing a new loss. If you have remaining questions, we are happy to answer them before the deadline.

Thank you for your time.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.