- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents theoretical analysis and empirical studies about the JEPA for text (T-JEPA). The motivation is that "although JEPA have been widely adopted in other modalities, its application to text is under-explored".
The authors tried to explain their contributions in Abstract and Introduction. However, if we take a look at Table 1, we see something very strange
- Except BERT, all baselines have an origin in Computer Vision (CV) domain, rather than in Natural Language Processing (NLP). This makes the evaluation very biased, and cannot reflect the state-of-the-art in NLP domain.
- BERT outperforms T-JEPA by an extremely huge margin, indicating that the text-JEPA is never working in this paper
- T-JEPA is either comparable to, or worse than other baselines, such as BYOL or VICReg Details of the above weaknesses will be discussed in the Weakness session.
I guess the authors may say "finding methods that do not work is also a contribution". However, I think this is only meaningful when the method is believed to be strong and we find some bugs of this strong method. Currently, all methods are apparently weaker than BERT, I don't think there is sufficient scientific insight.
Given the missing context in NLP domain, extremely weak performance in downstream tasks, as well as the fact that BERT can be interepreted as the discrete version of JEPA, I recommend a "clear reject" to this submission.
Critical strengths
None
Critical weaknesses
W1: BERT can be considered as the discrete version JEPA
Since JEPA is an evolving paradigm, I do not want to define this paradigm with strict criteria. However, I believe a general interpretation of JEPA is
- An latent encoder encodes raw input into a latent space. The procedure is often coupled with various regularizations to prevent latent mode collapse (e.g., the encoder maps all signals to the same embedding).
- Another module conducts mask reconstruction inside that regularized latent space.
Then let us take a look at the BERT
- A tokenizer encodes text signals into a set of finite indices, which we call vocabulary or codebook.
- Each index in the vocabulary is assigned with a high-dimentional embedding or vector. These embeddings are often randomly initialzed, and will be updated during the training.
- a Transformer will conduct mask reconstruction for the aforementioned finite set of embeddings
If we consider the discrete tokenization and the update of finite embeddings as a kind of regularization, then BERT is also conducting mask reconstruction task in a regularized space. Consequently, BERT can be considered as the discrete version of JEPA.
In this context, the claim that "JEPA is under-explored for text" itself becomes invalid.
W2: The paper is mostly written from a Computer Vision context, lacking up-to-date NLP contexts
BYOL or VICReg, to my knowledge, have an origin in CV domain.
The only NLP-derived encoder in this paper is BERT, completely ignoring other BERT variants, GPT-style variants, T5-style variants, as well as the CLIP-style text encoders.
I respect CV as a research field. Nevertheless, the ignoring of all these NLP baselines is inacceptable for this paper.
W3: Weak performance
The Table 1 shows that BERT largely outperform all other baselines. Moreover, other baselines can outperform T-JEPA in some datasets and some metrics.
Although the authors claim that they found the problems of T-JEPA, the weak performance suggests that the T-JEPA model is still not well trained, and there are more severe problems need to be addressed.
Questions for authors
None
3 comments
As mentioned, our work is a diagnostic study, where T-JEPA’s failure is the phenomenon we explain, not a method we propose. To rule out under-training, we rely on several controls. I-JEPA trains successfully under the same recipe, and BERT performs strongly with the same backbone, tokenizer and data, so the pipeline is sound. The failure follows the theory’s predicted order (MI saturation and elevated conditional variance before rank and cosine collapse), and this order holds across five seeds. BYOL with identical tuning collapses under masking but stays competitive under replacement, which points to conditional ambiguity rather than optimization as the cause. Shorter warmups and fixed learning rates make T-JEPA collapse earlier, not later. Finally, its chance-level accuracy with single-class F1 indicates a near-constant representation (cosine near 1), not a partially trained encoder.
Dear authors, I appreciate your polite and patient explanations. I respect your faith in this research topic.
Nevertheless, I do not think these responses address the critical concerns.
To ACs, I still don't think it is valuable to present a failure collection of CV-inspired yet NLP-targeted text representation learning methods in ICLR2027. Such failure collection can be presented in a blog, in a shorter paper, or in a workshop.
I will keep my original rating.
Why BERT Differs from JEPA, Why Our Baselines Are Appropriate, and Why T-JEPA Is Not Under-Trained
We appreciate the reviewer's perspective. However, we would like to clarify an important distinction between the two objectives.
[Response to W1]
In BERT, the prediction target is a discrete token from a finite vocabulary, and the model predicts a probability distribution over the vocabulary. In contrast, JEPA predicts a continuous target representation using a deterministic predictor trained with a squared-error objective (Eq. 5). Consequently, the optimal JEPA predictor is the conditional expectation of the target representation, which may not correspond to any valid target when multiple plausible continuations have substantially different representations. Therefore, although BERT can be viewed as masked prediction in a learned discrete representation space, its distributional prediction objective differs from the deterministic latent regression studied in our work.
Furthermore, Baevski et al. (2022) explicitly distinguish data2vec's contextualized latent targets from BERT's original token-level targets. Assran et al. (2023) also distinguish masked-token reconstruction methods from JEPA in their discussion of self-supervised learning paradigms.
Importantly, even under the reviewer's broader interpretation of JEPA, our central conclusion remains unchanged: distributional prediction over discrete targets can preserve multiple plausible alternatives, whereas deterministic regression to a single continuous target may suffer from the conditional-variance issue identified in our analysis. We also clarify that our claim is narrower than stating that JEPA is under-explored for text in general. Rather, we investigate the limitations of deterministic, squared-error latent prediction as a standalone objective for text encoders.
[Response to W2]
BYOL and VICReg indeed originate from the Computer Vision domain. However, our research question is why the JEPA paradigm, which originated in vision, cannot be directly transferred to text encoders, in the sense of BERT-style models, as it is in the original I-JEPA. These methods therefore serve as relevant non-contrastive self-supervised learning baselines, alongside BERT as a representative NLP pretraining objective.
Regarding GPT-style and T5-style models, our study deliberately focuses on how JEPA can be applied in the encoder-only scenario, to maintain a setting comparable to the original I-JEPA formulation. Moreover, causal language modeling and sequence denoising objectives predict distributions over discrete tokens or token sequences, which differ from the deterministic latent regression objective investigated in our work. We refer the reviewer to Appendix J for further motivation of our choice of baselines.
Regarding CLIP-style text encoders, CLIP is trained on paired image-text data with cross-modal contrastive supervision, i.e., it learns multimodal embeddings. This differs from our focus on single-modality self-supervised learning for text encoders, where the main question is why JEPA itself cannot be applied to text as it is to images.
We clarify that we did not ignore NLP baselines. As discussed in Appendix J, we consider LLM-JEPA and DLLM-JEPA, as well as BERT-JEPA, which combines the MLM loss with a JEPA loss. These approaches incorporate the JEPA objective only as an auxiliary loss alongside their primary objectives, such as causal language modeling, diffusion, or MLM. Removing these primary objectives leads to collapse or a substantial degradation in performance, so the JEPA objective does not stand alone in these works. This observation is consistent with our investigation of JEPA as a standalone objective, rather than as an auxiliary component of generative pretraining.
[Response to W3]
We would like to emphasize that our work does not aim to propose a new method or outperform existing NLP pretraining approaches. Instead, this is a diagnostic study that investigates the failure of deterministic JEPA when applied to text, in contrast to its successful application in vision. Our goal is to provide a theoretical and empirical understanding of this failure, which can serve as a foundation for future research on JEPA for text.
Regarding the concern that T-JEPA may simply be under-trained, our analysis identifies a specific theoretical limitation of the objective itself. In particular, we observe that the conditional structure of masked text prediction leads to elevated conditional variance, which is associated with the subsequent degeneration of learned representations. Therefore, the poor downstream performance is not merely an isolated empirical observation, but is consistent with the failure mechanism investigated in our work.