acceptodds
Withdrawn from ICLR 2027

LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language encoders are increasingly used as frozen backbones for multimodal language models and dense prediction, which consume the grid of patch tokens rather than a single pooled embedding. Image-text pretraining objectives, however, are defined on the pooled embedding, so the patch tokens are shaped only indirectly. We introduce LeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method. Image and text embeddings predict each other through modality-specific predictors with stop-gradient targets, and collapse is prevented by regularizing each modality with SIGReg. Thus, no negatives, temperature, momentum encoder, or teacher-student schedule are needed. On Datacomp-L, LeVLJEPA matches CLIP and SigLIP under linear probing and is more robust to background shifts, while its zero-shot accuracy is lower. On the patch tokens, which no objective supervises directly, it improves over the strongest contrastive baseline by more than mIoU on both semantic segmentation benchmarks. Used as the frozen visual backbone of a vision-language model, it obtains the highest accuracy on GQA, VQAv2, and POPE with two different language models. These results suggest that non-contrastive image-text pretraining is a viable way to learn visual features for dense prediction and multimodal language models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, representation learning, vision-language pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TKvpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LeVLJEPA, a non-contrastive vision-language pretraining objective that combines bidirectional cross-modal prediction and SIGReg distributional regularization for image and text embeddings. The central claim is that paired image-text pretraining does not require negatives. The results are similar to CLIP/InfoNCE and SigLIP under linear probing, while producing stronger patch-token features for linear semantic segmentation and frozen-backbone vision-language evaluation, but worse in zero-shot setting.

Critical strengths

The paper identifies a interesting mismatch between common pretraining objectives, which operate on pooled embeddings, and downstream systems that consume patch tokens. Evaluating dense features directly is a useful emphasis.

The objective is clear: bidirectional predictors with detached targets plus per-modality SIGReg. This makes it a clean recipe, without in-batch negatives, or intricate parameter optimizations.

The compared baselines InfoNCE, SigLIP, and LeVLJEPA use the same ViT-B/16, data pool, samples seen, and downstream protocols. This makes the results informative.

The ablation indicates that direct MSE collapses, SIGReg alone raises rank without yielding useful representations, and predictor/stop-gradient without SIGReg is insufficient. This supports the method.

Critical weaknesses

VL-JEPA is discussed as the closest method, but it is not evaluated. This is a major omission because the proposed method is positioned as the non-contrastive realization of a direction explicitly raised by VL-JEPA. Even if exact scale matching is expensive, a controlled CC12M comparison or other comparison is essential.

The related-work discussion focuses mainly on CLIP, SigLIP, VL-JEPA, BYOL, I-JEPA, DINO, and LeJEPA. That is not enough for a strong 'first fully non-contrastive end-to-end vision-language pretraining' claim. I wonder if the claim is correct. Narrowing down the claim seems needed, e.g. first joint-embedding VL pretraining that is fully free of negatives and of teacher/EMA machinery, trained end-to-end from scratch.

There is no image-text retrieval evaluation, which is a standard and direct measure of cross-modal alignment and would clarify what information is lost relative to contrastive training, or what is gained.

The method figure doesn't make clear what is the novel element and its merit inside this JEPA formulation.

Questions for authors

What are other vision-language methods without explicit negatives, and how does LeVLJEPA differ technically?

Can you include a controlled VL-JEPA baseline, preferably on CC12M and, if feasible, DataComp-L?

Why is VL retrieval omitted?

Do the gains hold for larger backbones and more patches?

Reviewer_pNMwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a non-contrastive end-to-end vision-language pretraining method that does not use any negative pairs. They use modality-specific predictors with stop-gradient targets to predict the embeddings of the other modality. To prevent each modality’s embeddings from collapsing, they regularize them with SIGReg, which constrains the embeddings to follow an isotropic Gaussian. The authors demonstrate that the proposed encoder achieves better performance than CLIP and SigLIP on tasks that use patch tokens, such as semantic segmentation and VQA, but worse performance on zero-shot classification, while achieving on-par performance on linear probing.

Critical strengths

The paper is easy to read and understand. It provides an interesting approach to non-contrastive vision-language pretraining that does not require any negatives, temperature, momentum encoder or teacher-student schedule. The authors show consistent gains on dense prediction and frozen-backbone VQA, suggesting that the method learns better patch-level features than contrastive approaches.

Critical weaknesses

However, the paper has several key weaknesses listed below:

  • The claim of the authors to be the “the first fully non-contrastive end-to-end vision-language pretraining method” is overstated and incorrect. A previous work nCLIP (Zhou et al., CVPR 2023) has explored training image-text encoders end-to-end in a non-contrastive manner without negatives and has reported similar trade-offs as the authors like weaker zero-shot and better representations. Furthermore, many captioning based methods like CapPa (Tschannen et al., Neurips 2023), LocCa (Wan et al, Neurips 2024) and AIMv2 (Fini, et al., CVPR 2025) are non-contrastive.

  • In my opinion, the motivation of the paper does not flow logically. One of its key motivations is that downstream tasks extract information from the patch tokens, while vision-language pretraining applies contrastive learning to the global pooled embedding. However, LeVLJEPA’s loss is also applied only to the pooled [CLS] embedding. Moreover, whether an objective is contrastive and whether it supervises patch tokens seem to be largely orthogonal choices, as shown by prior works such as SigLIP 2 (Tschannen et al., arXiv 2025), which combines a contrastive objective with patch-level supervision. Similarly, FILIP (Yao et al., ICLR 2022) and SPARC (Bica et al., ICML 2024) apply contrastive losses at the patch level.

  • Although the paper provides some empirical evidence that the approach leads to better patch features (from the segmentation and VQA experiments), the authors do not explain why a non-contrastive pooled objective should yield better patch features than a contrastive pooled objective, or than approaches that combine contrastive learning with patch-level supervision.

  • As mentioned earlier, several image-text pretraining approaches, such as SigLIP 2, FILIP and SPARC, supervise patch tokens directly alongside a contrastive objective. The paper neither cites nor discusses them, and the authors evaluate only against CLIP and SigLIP. At a minimum, these approaches should be acknowledged and discussed; ideally, at least one patch-supervised baseline should be included under matched data and compute.

Questions for authors

I would appreciate it if the authors could address the points regarding motivation and overstated novelty in the weaknesses section. In particular:

  • What explains improved patch features given that the proposed loss is applied on the pooled [CLS] embedding?

  • How does LeVLJEPA compare with prior non-contrastive methods (e.g., nCLIP) and patch-supervised methods (e.g., SigLIP 2, SPARC)?

  • Is it possible to compare against at least one of them like SigLIP 2 or nCLIP?

Reviewer_FSbQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper presents LeVLJEPA, a non-contrastive pretraining approach for learning CLIP style image + text eoncoders. The paper a ViT base image encoder and a GPT-2 text encoder, similar to CLIP base, without contrastive negatives. Instead of InfoNCE or a sigmoid loss, each modality has a small MLP predictor that regresses the other modality's embedding.To prevent collapse, each modality's embedding distribution is pushed toward an isotropic Gaussian with SIGReg, the regularizer from LeJEPA.

On DataCompL (comprising of 92M image+text training pairs) the method matches CLIP and SigLIP under linear probing and is worse at zero-shot classification. Its patch tokens are better, giving about +2 mIoU on linear segmentation and the best frozen-backbone VQA scores on GQA, VQAv2 and POPE with two small LLMs.

Critical strengths
  1. Frozen VL encoders are mostly consumed through their patch tokens (LLaVA style VLMs, dense prediction, robotics), yet they are compared on pooled embedding metrics. The paper's observation that zero-shot ranking reverses on dense and VLM evaluations (SigLIP is best zero-shot but the weakest backbone) is worth sharing.
  2. Controlled comparison: All three methods (CLIP, SigLIP, proposed method) share the backbone, data, samples seen and evaluation code, and zero shot uses a single prompt template. This makes the comparison across the three methods fair.
  3. Consistent dense gains: proposed method wins on two segmentation datasets and on 6 of 6 VLM benchmark × LLM combinations.
Critical weaknesses
  1. Paper claims that "LeVLJEPA, the first fully non-contrastive end-to-end vision-language pretraining method". This claim may not be fully accurate. See Non-Contrastive Learning Meets Language-Image Pre-Training (https://arxiv.org/abs/2210.09304).
  2. Limited novelty: approach combines SimSiam style predictor with LeJEPA's SIGReg, applied across modalities. VL-JEPA already suggests swapping InfoNCE for a sample independent anti collapse regularizer.
  3. baselines may be handicapped: Contrastive methods are known to depend on batch size, yet everything runs at a global batch of 4096. OpenCLIP's DataCompL recipes use much larger batches. ImageNet zero-shot of 47% for InfoNCE and 51% for SigLIP is far below typical DataCompL ViT-B/16 numbers (around 60%+).
  4. several key comparisons are missing: a) Dense-feature methods: there is no comparison with vision only SSL methods, which are the natural patch-feature baselines (DINOv2, I-JEPA, LeJEPA etc), or with dense aware VL objectives (like SigLIP 2). b) VL-JEPA: is not run as a baseline, even though it is the closest prior work, c) Retrieval performance: no image-text retrieval on COCO or Flickr datasets, which are standard VL evaluations.
Questions for authors

Please share your responses for 1-4. Thanks !

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.