- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LeVLJEPA, a non-contrastive vision-language pretraining objective that combines bidirectional cross-modal prediction and SIGReg distributional regularization for image and text embeddings. The central claim is that paired image-text pretraining does not require negatives. The results are similar to CLIP/InfoNCE and SigLIP under linear probing, while producing stronger patch-token features for linear semantic segmentation and frozen-backbone vision-language evaluation, but worse in zero-shot setting.
Critical strengths
The paper identifies a interesting mismatch between common pretraining objectives, which operate on pooled embeddings, and downstream systems that consume patch tokens. Evaluating dense features directly is a useful emphasis.
The objective is clear: bidirectional predictors with detached targets plus per-modality SIGReg. This makes it a clean recipe, without in-batch negatives, or intricate parameter optimizations.
The compared baselines InfoNCE, SigLIP, and LeVLJEPA use the same ViT-B/16, data pool, samples seen, and downstream protocols. This makes the results informative.
The ablation indicates that direct MSE collapses, SIGReg alone raises rank without yielding useful representations, and predictor/stop-gradient without SIGReg is insufficient. This supports the method.
Critical weaknesses
VL-JEPA is discussed as the closest method, but it is not evaluated. This is a major omission because the proposed method is positioned as the non-contrastive realization of a direction explicitly raised by VL-JEPA. Even if exact scale matching is expensive, a controlled CC12M comparison or other comparison is essential.
The related-work discussion focuses mainly on CLIP, SigLIP, VL-JEPA, BYOL, I-JEPA, DINO, and LeJEPA. That is not enough for a strong 'first fully non-contrastive end-to-end vision-language pretraining' claim. I wonder if the claim is correct. Narrowing down the claim seems needed, e.g. first joint-embedding VL pretraining that is fully free of negatives and of teacher/EMA machinery, trained end-to-end from scratch.
There is no image-text retrieval evaluation, which is a standard and direct measure of cross-modal alignment and would clarify what information is lost relative to contrastive training, or what is gained.
The method figure doesn't make clear what is the novel element and its merit inside this JEPA formulation.
Questions for authors
What are other vision-language methods without explicit negatives, and how does LeVLJEPA differ technically?
Can you include a controlled VL-JEPA baseline, preferably on CC12M and, if feasible, DataComp-L?
Why is VL retrieval omitted?
Do the gains hold for larger backbones and more patches?