- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Tri-JEPA, an interaction-aware joint-embedding predictive architecture for self-supervised multimodal pretraining. The method uses three predictive objectives, shared, unique, and synergistic, to capture different types of interactions between modalities. The authors evaluate the method on several downstream tasks using image-based modalities and show that it outperforms the selected baselines.
Critical strengths
-
The proposed method shows good performance and outperforms the selected baselines.
-
The authors provide ablation studies that support the effectiveness of using all three predictive objectives.
-
The paper is well-written and easy to follow.
Critical weaknesses
-
The novelty of the proposed method is limited. Previous works such as FactorCL (Liang et al., 2023), CoMM (Dufumier et al., 2024), and InfMasking (Wen et al., 2025) have already explored learning shared, unique, and synergistic information in multimodal self-supervised learning. JEPA-based multimodal representation prediction has also been explored in prior work such as VL-JEPA (Chen et al., 2025). The proposed method mainly combines these existing ideas using three predictive objectives.
-
It is unclear whether the three predictive objectives actually encourage the model to learn the intended types of information. For example, the unique objective predicts masked patches using visible patches from the same modality, but this does not necessarily mean that the learned information is unique to that modality, since it could also be shared with other modalities. Similarly, the synergistic objective uses multiple modalities jointly to predict a target modality, but the model could simply combine information from each source independently without learning interactions that can only be captured when the modalities are considered together. Therefore, the connection between the proposed objectives and the three information types is not sufficiently justified.
-
The analysis in Section 5.1 provides limited evidence that the model actually learns synergistic information. The authors use the difference between joint mutual information and the sum of individual mutual information values to support this claim. However, this measure reflects the difference between synergy and redundancy, rather than directly measuring synergy. A positive value alone does not show how much synergistic information is learned. In addition, the authors do not clearly explain how mutual information is estimated or provide uncertainty estimates for the reported values. This makes it difficult to assess whether the results reliably support the claim that the synergistic objective encourages the intended type of interaction.
Questions for authors
-
The authors claim that Tri-JEPA achieves better performance using fewer pretraining epochs and less data. However, the proposed method optimizes multiple predictive objectives at each training step, which may increase computational cost. Could the authors provide a comparison of training time, GPU-hours, or FLOPs against the baselines?
-
The main results in Table 1 are reported only as averages across datasets rather than showing the performance on each dataset separately. This is a limitation of the experimental evaluation. Could the authors provide the individual dataset results for all downstream tasks to clarify whether the improvements are consistent across datasets or mainly driven by a subset of them?
-
Although the method is presented as a general framework for multimodal pretraining, the experiments are limited to image-based modalities. How well would the proposed objectives generalize to other types of modalities, such as text and audio?
4 comments
Q3. Generalization beyond original settings
We thank the reviewer for this important question and find encouraging results.
Heterogeneous modalities (vision-language).
We pretrained Tri-JEPA on BigEarthNet.txt [1] using S1, S2, and text as three modalities. All three modalities share the same ViT-B encoder and retain the same pretraining protocol as our main experiments. We then evaluated Tri-JEPA representations on the BigEarthNet.txt benchmark, following its fine-tuning protocol. We project the resulting Tri-JEPA tokens into InternVL3-1B and fine-tune the model. As shown below, Tri-JEPA improves captioning and question answering over the same language model without sensor tokens and outperforms the published baseline.
Table 4. BigEarthNet.txt benchmark with an LLM using Tri-JEPA tokens.
| Model | Captioning BLEU-4 | Binary questions | Multiple choice |
|---|---|---|---|
| RS-InternVL | 0.340 | 0.733 | 0.515 |
| InternVL3-1B (w/o Tri-JEPA) | 0.277 | 0.643 | 0.540 |
| InternVL3-1B (w/ Tri-JEPA) | 0.494 | 0.781 | 0.804 |
Heterogeneous modalities (more results).
We further evaluated Tri-JEPA on additional heterogeneous-modality benchmarks: CLEVR [2], VQA 2.0 [3], GQA [4], and LLP [5]. We pretrained Tri-JEPA on each benchmark and evaluated the learned representations using linear probes with frozen encoders. We compare against FactorCL, CoMM, and InfMasking using their official implementations. As shown below, Tri-JEPA consistently achieves the best performance across all benchmarks.
Table 5. Heterogeneous-modality benchmarks via linear probes.
| Model | CLEVR (Image+Text) | VQA 2.0 (Image+Text) | GQA (Image+Text) | LLP (Audio+Image+Motion) |
|---|---|---|---|---|
| FactorCL | 0.353 | 0.338 | 0.314 | -- |
| CoMM | 0.415 | 0.370 | 0.343 | 0.787 |
| InfMasking | 0.394 | 0.365 | 0.337 | 0.749 |
| Tri-JEPA | 0.448 | 0.381 | 0.388 | 0.827 |
References
[1] Herzog et al. (2026). BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation. arXiv:2603.29630.
[2] Johnson et al. (2017). CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CVPR.
[3] Goyal et al. (2017). Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR.
[4] Hudson and Manning (2019). GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. CVPR.
[5] Tian et al. (2020). Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing. ECCV.
Kind Follow-up
Dear Reviewer,
We wanted to kindly follow up and see if you have any additional questions or concerns regarding our response. We would be happy to provide further clarification if helpful.
Thank you.
Thank you for the additional experiments and clarifications. I appreciate the authors' efforts, and several of my original concerns have been addressed.
However, I still find novelty and contributions limited. I am also not fully convinced that the objectives encourage the intended types of information. The controlled experiment is helpful, but the evidence on real data remains limited. The response also does not address my concern of how mutual information was estimated and its limitations.
The additional results strengthen the paper, but my main concerns remain. I will maintain my original rating of 2.
W1. Positioning of Tri-JEPA
We thank the reviewer for highlighting these related works. While they share the broader goal of learning multimodal information interactions, Tri-JEPA is the first to explicitly formulate shared, unique, and synergistic interactions as three distinct predictive objectives within a JEPA-based framework. Unlike FactorCL, CoMM, and InfMasking, which learn interactions through contrastive objectives on global/augmented embeddings, Tri-JEPA predicts the same masked target in latent space from different source sets: the target modality itself, another modality, or multiple modalities jointly. This also distinguishes Tri-JEPA from VL-JEPA, which focuses on vision-to-text embedding prediction rather than learning distinct information interactions. Our formulation allows us to independently control each objective and examine its contribution to representation learning and downstream transfer. Kindly find our further evidence for the intended interactions, efficiency, and generalizability of Tri-JEPA.
W2 & W3. Evidence for the intended interactions
The reviewer's observation is on-point, and we agree that further evidence would help establish how the objectives encourage the intended interactions. We provide additional results below.
Controlled experiment with known information structure.
To examine whether the objectives encourage their intended interactions, we constructed a controlled dataset where each sample consists of three inputs, a, b, and c, each represented by a sequence of tokens. Each input contains the same shared variable s, its own unique variable ui, and a synergistic variable xi (i∈{a,b,c}). We independently sample s, ua, ub, uc, xa, and xb uniformly over four classes and define xc=−(xa+xb)mod4 (i.e., the non-negative remainder after division by 4). By construction, s is shared across all three inputs, each ui is unique to its corresponding input, and predicting xc requires combining xa and xb, as neither variable alone provides information about xc.
We pretrained Tri-JEPA and its single-objective variants on 50,000 samples using the same training protocol and evaluated them on 3,000 held-out samples. We used linear probes on frozen encoder representations to assess shared and unique information. To assess synergy, we evaluated the prediction of xc using the two inputs jointly. The results below provide further evidence that each objective encourages its intended interaction while Tri-JEPA emphasizes all three.
Table 1. Prediction accuracy for shared, unique, and synergistic variables on controlled data.
Q1. Compute and data efficiency.
Compute efficiency.
We thank the reviewer for this question. Under identical settings, Tri-JEPA requires substantially less computation per step than the baselines. FactorCL, CoMM, and InfMasking encode two augmented views of each sample. CoMM and InfMasking additionally run a fusion transformer for every modality subset, and InfMasking further processes five masked copies of each subset. In contrast, Tri-JEPA encodes the visible tokens from a single view and reuses them across all three objectives. Each objective simply passes a different combination of the encoded tokens to a lightweight predictor. This avoids repeated encoding and substantially reduces computation.
Table 2. Pretraining computation under identical settings.
Data efficiency.
We evaluated data efficiency using subsets of pretraining data. With only half of the data, Tri-JEPA outperforms many of the baselines pretrained on the full dataset.
Table 3. Tri-JEPA pretrained on subsets.
Q2. Per-dataset results.
Please find the per-dataset results in the appendix. Tri-JEPA achieves the best average rank of 1.07 across all datasets, demonstrating its consistent benefits.