acceptodds
Under review as a conference paper at ICLR 2027

Tri-JEPA: Interaction-Aware Self-Supervised Multimodal Pretraining

Abstract

Self-supervised multimodal pretraining aims to learn representations from unlabeled multimodal data that transfer across downstream tasks. Different tasks, however, may rely on different multimodal interactions: information shared across modalities, information unique to a single modality, and synergistic information that emerges only when modalities are considered jointly. Since the downstream task is unknown at pretraining time, encouraging all three may yield a more broadly useful prior. Existing methods either emphasize only a subset of these interactions or capture them implicitly through objectives over fused multimodal representations. We introduce Tri-JEPA, an interaction-aware joint-embedding predictive architecture with three predictive objectives defined by different source-target configurations. For a target modality, the shared objective predicts its representation from another modality, the unique objective predicts it from the same modality, and the synergistic objective predicts it from all other modalities jointly. These assignments are cycled across modalities for balanced coverage. The tailored objectives enable us to examine how different multimodal interactions are encouraged during pretraining and how their contributions vary across downstream tasks. Under controlled settings, Tri-JEPA consistently outperforms the multimodal self-supervised baselines considered across diverse downstream tasks. Overall, our results show that interaction-aware predictive objectives are a useful design principle for self-supervised multimodal pretraining.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Earth Observation, Self-supervised Learning, Multimodal Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_77wgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Tri-JEPA, an interaction-aware joint-embedding predictive architecture for self-supervised multimodal pretraining. The method uses three predictive objectives, shared, unique, and synergistic, to capture different types of interactions between modalities. The authors evaluate the method on several downstream tasks using image-based modalities and show that it outperforms the selected baselines.

Critical strengths
  1. The proposed method shows good performance and outperforms the selected baselines.

  2. The authors provide ablation studies that support the effectiveness of using all three predictive objectives.

  3. The paper is well-written and easy to follow.

Critical weaknesses
  1. The novelty of the proposed method is limited. Previous works such as FactorCL (Liang et al., 2023), CoMM (Dufumier et al., 2024), and InfMasking (Wen et al., 2025) have already explored learning shared, unique, and synergistic information in multimodal self-supervised learning. JEPA-based multimodal representation prediction has also been explored in prior work such as VL-JEPA (Chen et al., 2025). The proposed method mainly combines these existing ideas using three predictive objectives.

  2. It is unclear whether the three predictive objectives actually encourage the model to learn the intended types of information. For example, the unique objective predicts masked patches using visible patches from the same modality, but this does not necessarily mean that the learned information is unique to that modality, since it could also be shared with other modalities. Similarly, the synergistic objective uses multiple modalities jointly to predict a target modality, but the model could simply combine information from each source independently without learning interactions that can only be captured when the modalities are considered together. Therefore, the connection between the proposed objectives and the three information types is not sufficiently justified.

  3. The analysis in Section 5.1 provides limited evidence that the model actually learns synergistic information. The authors use the difference between joint mutual information and the sum of individual mutual information values to support this claim. However, this measure reflects the difference between synergy and redundancy, rather than directly measuring synergy. A positive value alone does not show how much synergistic information is learned. In addition, the authors do not clearly explain how mutual information is estimated or provide uncertainty estimates for the reported values. This makes it difficult to assess whether the results reliably support the claim that the synergistic objective encourages the intended type of interaction.

Questions for authors
  1. The authors claim that Tri-JEPA achieves better performance using fewer pretraining epochs and less data. However, the proposed method optimizes multiple predictive objectives at each training step, which may increase computational cost. Could the authors provide a comparison of training time, GPU-hours, or FLOPs against the baselines?

  2. The main results in Table 1 are reported only as averages across datasets rather than showing the performance on each dataset separately. This is a limitation of the experimental evaluation. Could the authors provide the individual dataset results for all downstream tasks to clarify whether the improvements are consistent across datasets or mainly driven by a subset of them?

  3. Although the method is presented as a general framework for multimodal pretraining, the experiments are limited to image-based modalities. How well would the proposed objectives generalize to other types of modalities, such as text and audio?

4 comments

W1. Positioning of Tri-JEPA

We thank the reviewer for highlighting these related works. While they share the broader goal of learning multimodal information interactions, Tri-JEPA is the first to explicitly formulate shared, unique, and synergistic interactions as three distinct predictive objectives within a JEPA-based framework. Unlike FactorCL, CoMM, and InfMasking, which learn interactions through contrastive objectives on global/augmented embeddings, Tri-JEPA predicts the same masked target in latent space from different source sets: the target modality itself, another modality, or multiple modalities jointly. This also distinguishes Tri-JEPA from VL-JEPA, which focuses on vision-to-text embedding prediction rather than learning distinct information interactions. Our formulation allows us to independently control each objective and examine its contribution to representation learning and downstream transfer. Kindly find our further evidence for the intended interactions, efficiency, and generalizability of Tri-JEPA.

W2 & W3. Evidence for the intended interactions

The reviewer's observation is on-point, and we agree that further evidence would help establish how the objectives encourage the intended interactions. We provide additional results below.

Controlled experiment with known information structure.

To examine whether the objectives encourage their intended interactions, we constructed a controlled dataset where each sample consists of three inputs, aa, bb, and cc, each represented by a sequence of tokens. Each input contains the same shared variable ss, its own unique variable uiu_i, and a synergistic variable xix_i (i∈{a,b,c}i \in \{a,b,c\}). We independently sample ss, uau_a, ubu_b, ucu_c, xax_a, and xbx_b uniformly over four classes and define xc=−(xa+xb) mod 4x_c = -(x_a + x_b) \bmod 4 (i.e., the non-negative remainder after division by 4). By construction, ss is shared across all three inputs, each uiu_i is unique to its corresponding input, and predicting xcx_c requires combining xax_a and xbx_b, as neither variable alone provides information about xcx_c.

We pretrained Tri-JEPA and its single-objective variants on 50,000 samples using the same training protocol and evaluated them on 3,000 held-out samples. We used linear probes on frozen encoder representations to assess shared and unique information. To assess synergy, we evaluated the prediction of xcx_c using the two inputs jointly. The results below provide further evidence that each objective encourages its intended interaction while Tri-JEPA emphasizes all three.

Table 1. Prediction accuracy for shared, unique, and synergistic variables on controlled data.

ModelSharedUniqueSynergy
Chance0.2500.2500.250
Shared only0.9990.3270.251
Unique only0.3510.9990.248
Synergistic only0.2960.2521.000
Tri-JEPA0.9940.9400.999

Q1. Compute and data efficiency.

Compute efficiency.

We thank the reviewer for this question. Under identical settings, Tri-JEPA requires substantially less computation per step than the baselines. FactorCL, CoMM, and InfMasking encode two augmented views of each sample. CoMM and InfMasking additionally run a fusion transformer for every modality subset, and InfMasking further processes five masked copies of each subset. In contrast, Tri-JEPA encodes the visible tokens from a single view and reuses them across all three objectives. Each objective simply passes a different combination of the encoded tokens to a lightweight predictor. This avoids repeated encoding and substantially reduces computation.

Table 2. Pretraining computation under identical settings.

ModelTFLOPs / stepRelative compute
FactorCL56.03.6×
CoMM62.44.1×
InfMasking186.212.1×
Tri-JEPA15.41.0×

Data efficiency.

We evaluated data efficiency using subsets of pretraining data. With only half of the data, Tri-JEPA outperforms many of the baselines pretrained on the full dataset.

Table 3. Tri-JEPA pretrained on subsets.

Pretraining dataClassificationSegmentation
25%0.7310.625
50%0.7490.642
100%0.7570.662

Q2. Per-dataset results.

Please find the per-dataset results in the appendix. Tri-JEPA achieves the best average rank of 1.07 across all datasets, demonstrating its consistent benefits.

Q3. Generalization beyond original settings

We thank the reviewer for this important question and find encouraging results.

Heterogeneous modalities (vision-language).

We pretrained Tri-JEPA on BigEarthNet.txt [1] using S1, S2, and text as three modalities. All three modalities share the same ViT-B encoder and retain the same pretraining protocol as our main experiments. We then evaluated Tri-JEPA representations on the BigEarthNet.txt benchmark, following its fine-tuning protocol. We project the resulting Tri-JEPA tokens into InternVL3-1B and fine-tune the model. As shown below, Tri-JEPA improves captioning and question answering over the same language model without sensor tokens and outperforms the published baseline.

Table 4. BigEarthNet.txt benchmark with an LLM using Tri-JEPA tokens.

ModelCaptioning BLEU-4Binary questionsMultiple choice
RS-InternVL0.3400.7330.515
InternVL3-1B (w/o Tri-JEPA)0.2770.6430.540
InternVL3-1B (w/ Tri-JEPA)0.4940.7810.804

Heterogeneous modalities (more results).

We further evaluated Tri-JEPA on additional heterogeneous-modality benchmarks: CLEVR [2], VQA 2.0 [3], GQA [4], and LLP [5]. We pretrained Tri-JEPA on each benchmark and evaluated the learned representations using linear probes with frozen encoders. We compare against FactorCL, CoMM, and InfMasking using their official implementations. As shown below, Tri-JEPA consistently achieves the best performance across all benchmarks.

Table 5. Heterogeneous-modality benchmarks via linear probes.

ModelCLEVR (Image+Text)VQA 2.0 (Image+Text)GQA (Image+Text)LLP (Audio+Image+Motion)
FactorCL0.3530.3380.314--
CoMM0.4150.3700.3430.787
InfMasking0.3940.3650.3370.749
Tri-JEPA0.4480.3810.3880.827

References

[1] Herzog et al. (2026). BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation. arXiv:2603.29630.

[2] Johnson et al. (2017). CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CVPR.

[3] Goyal et al. (2017). Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR.

[4] Hudson and Manning (2019). GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. CVPR.

[5] Tian et al. (2020). Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing. ECCV.

AuthorsOpenReview ↗

Kind Follow-up

Dear Reviewer,

We wanted to kindly follow up and see if you have any additional questions or concerns regarding our response. We would be happy to provide further clarification if helpful.

Thank you.

Thank you for the additional experiments and clarifications. I appreciate the authors' efforts, and several of my original concerns have been addressed.

However, I still find novelty and contributions limited. I am also not fully convinced that the objectives encourage the intended types of information. The controlled experiment is helpful, but the evidence on real data remains limited. The response also does not address my concern of how mutual information was estimated and its limitations.

The additional results strengthen the paper, but my main concerns remain. I will maintain my original rating of 2.

Reviewer_v5D2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Tri-JEPA, a self-supervised pretraining method for multi-sensor satellite imagery. The motivation is that different downstream tasks may rely on information that is shared between modalities, specific to one modality, or only available when modalities are combined. Existing methods such as CoMM and InfMasking learn all of these together in a single fused representation. Tri-JEPA uses three JEPA-style prediction tasks: the masked patches of one modality are predicted from another modality, from the same modality, or from all the other modalities together, and the three losses are added up. The authors pretrain a ViT-B on 130K locations with Sentinel-1, Sentinel-2 and WorldView images, retrain eight baselines on the same data, and report that Tri-JEPA performs best on classification, segmentation, detection and change detection with a frozen backbone.

Critical strengths

The method is simple and beats all eight baselines on four different task types, with a clear margin on detection (0.869 vs. 0.780 for DINO), and it stays ahead in the low-label setting (Fig. 6).

Critical weaknesses
  1. The method is close to existing multimodal JEPA work. It is basically I-JEPA with the visible patches taken from other modalities, and AnySat already uses a JEPA objective on multi-sensor EO data; the main difference is three prediction tasks instead of one. The paper does not explain why this gives such a large gain, and the ablation does not answer it: in Fig. 5, dropping any one loss costs at most about 2 points on classification and segmentation, while Tri-JEPA is 13 points above AnySat on classification. A single JEPA objective on merged tokens with the same code is missing.

  2. The MI analysis is a useful sanity check, but it does not show that each loss learns the kind of information it is named after. It is computed on the learned features, so it mostly reflects how each model was trained. For example, a model trained only to predict T from (X1, X2) can get Δ > 0 even if the data has no synergy. A check on data where shared, unique and synergistic information are known (e.g. Trifeature) would make the claim much stronger.

  3. All results are on one pretraining dataset. Also, contribution 3 says Tri-JEPA uses "fewer pretraining epochs and less data", but §4.3 says all methods use the same data.

Questions for authors
  1. Can you show per-dataset results for Table 1?
  2. Can you add a baseline with your own code (same loss, masking and augmentation) but a single JEPA objective on merged tokens? Downstream results for the three single-loss variants would also help (Fig. 4 only shows retrieval). This would show how much of the gain really comes from splitting the loss.
  3. Which MI estimator did you use in §5.1, and can you add a check on data where the amount of shared, unique and synergistic information is known?
5 comments

W1. Merged-token JEPA control

We thank the reviewer for this question. We provide the requested merged-token JEPA baseline and additional analysis to explain why separating the prediction objectives improves performance.

Merged-token JEPA baseline.

We pretrained the requested baseline (i.e., Merged-token JEPA) using a single JEPA objective that predicts masked target tokens from the merged visible tokens of all three modalities. Merged-token JEPA uses the same code, ViT-B architecture, masking, augmentations, pretraining data and epochs as Tri-JEPA. We evaluated both InfoNCE and smooth L1 losses to distinguish the effects of the prediction objectives from the loss function. As shown in Table 1, Tri-JEPA consistently improves task performance. The improvements remain substantial with smooth L1, indicating that separating the prediction contexts matters beyond the choice of loss.

Table 1. JEPA controls on GEO-Bench.

ModelLossClassificationSegmentation
Merged-token JEPASmooth L10.6650.477
Tri-JEPASmooth L10.7540.659
Merged-token JEPAInfoNCE0.6820.475
Tri-JEPAInfoNCE0.7570.662

Why separating the prediction contexts helps.

The merged-token predictor can predict masked patches using the target sensor's own visible patches without learning cross-sensor relationships. To examine this behavior, we analyzed its attention when predicting masked RGB tokens across 1,024 tiles and 64 masks per target (Table 2). Both merged-token controls assign nearly all attention to visible RGB tokens despite having access to S1 and S2. This suggests that they rely primarily on within-sensor information. In contrast, Tri-JEPA excludes the target sensor from its shared and synergistic prediction contexts, promoting cross-sensor prediction.

Table 2. Attention distribution when predicting masked RGB tokens.

ModelRGB (target)S1S2
Merged-token JEPA0.9990.0000.001
Tri-JEPA--0.3100.690

W2. Evidence for the intended interactions

The reviewer's observation is on-point, and we agree that further evidence would help establish how the objectives encourage the intended interactions. We provide additional results below.

Controlled experiment with known information structure.

To examine whether the objectives encourage their intended interactions, we constructed a controlled dataset where each sample consists of three inputs, aa, bb, and cc, each represented by a sequence of tokens. Each input contains the same shared variable ss, its own unique variable uiu_i, and a synergistic variable xix_i (i∈{a,b,c}i \in \{a,b,c\}). We independently sample ss, uau_a, ubu_b, ucu_c, xax_a, and xbx_b uniformly over four classes and define xc=−(xa+xb) mod 4x_c = -(x_a + x_b) \bmod 4 (i.e., the non-negative remainder after division by 4). By construction, ss is shared across all three inputs, each uiu_i is unique to its corresponding input, and predicting xcx_c requires combining xax_a and xbx_b, as neither variable alone provides information about xcx_c.

We pretrained Tri-JEPA and its single-objective variants on 50,000 samples using the same training protocol and evaluated them on 3,000 held-out samples. We used linear probes on frozen encoder representations to assess shared and unique information. To assess synergy, we evaluated the prediction of xcx_c using the two inputs jointly. The results below provide further evidence that each objective encourages its intended interaction while Tri-JEPA emphasizes all three.

Table 3. Prediction accuracy for shared, unique, and synergistic variables on controlled data.

ModelSharedUniqueSynergy
Chance0.2500.2500.250
Shared only0.9990.3270.251
Unique only0.3510.9990.248
Synergistic only0.2960.2521.000
Tri-JEPA0.9940.9400.999

Thank you for your detailed response.

Reviewer_vLAgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that multimodal pretraining should encourage shared, unique and synergistic interactions, since downstream tasks may rely on any of them. Tri-JEPA predicts masked patches of a target modality from another modality, from the same modality, or from all other modalities jointly, cycling the target modality across steps. A ViT-Base is pretrained on three satellite image modalities. The paper analyzes the variants with mutual-information estimates and tests frozen transfer on four task families against eight baselines.

Critical strengths
  1. The idea is simple and clean. All three objectives come from one JEPA by changing only the source set, and cycling keeps the per-step cost fixed. It is easy to adopt and ablate.
  2. The downstream comparison is controlled and broad. Baselines are said to share the backbone and corpus, and results use 3 seeds. Tri-JEPA has the best mean in every column of Table 1, with large margins for classification and detection (0.869 vs. 0.780 for DINO). This is solid support if the comparison is fair (see Weakness 2).
Critical weaknesses
  1. The evidence that each objective induces its "intended" interaction, the main conceptual claim, is thin. The objectives do not isolate PID atoms, as the paper concedes. So the claim rests on Sec. 5.1, where the MI estimator is never named. The estimates come without uncertainty, a shuffled-pair control or seed repetition, and Δ\Delta is missing for the full model. Such estimates depend on the estimator [1]. The near-zero cross-modal MI of the unique-only model (0.050) is surprising for a shared encoder over co-located sensors and may reflect embedding alignment rather than information content. Finally, the text says removing synergy gives the largest classification drop, but in Fig. 5 removing the unique objective is clearly worse. The "task-dependent contribution" conclusion thus rests on small, partly misreported differences.
  2. It is unclear what drives the gains, and fairness cannot be checked. In Fig. 5, removing any single objective still leaves detection close to the full model and well above every baseline in Table 1. So much of the detection margin may come from other choices, such as the InfoNCE loss (I-JEPA [2] uses an L2L_2 loss) or the augmentations. Downstream results for the single-objective variants, the right control, are missing. Baseline tuning and multimodal adaptation are not described. The claim of using "less data" contradicts the same-data protocol. CoMM and FactorCL also fall far below unimodal DINO on segmentation, so the reimplementations may be under-tuned.
  3. The downstream protocol is under-specified. Inputs differ across benchmarks, but the paper does not say which patch projection handles unfamiliar channels or whether modalities are ever fused downstream. Without this, the downstream "synergy" contribution in Fig. 5 cannot be read, since synergy needs jointly observed modalities. Table 1 also gives only cross-dataset averages.
  4. The evidence is narrower than the claims. The paper presents interaction-aware objectives as a general principle for multimodal SSL. Yet all results use one US corpus, three co-registered image modalities, one backbone and frozen evaluation. The image-only limitation is acknowledged, but nothing covers heterogeneous modalities or larger scale. This limits significance for the broader ICLR audience.

References

[1] Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, and Louis-Philippe Morency. "Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework." Advances in Neural Information Processing Systems (NeurIPS), 2023.

[2] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. "Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture." IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

Questions for authors
  1. Which MI estimator is used, and is it invariant to non-aligned embedding spaces? Could you add uncertainty, a shuffled-pair control and the missing Δ\Delta values, and reconcile the ablation text with Fig. 5?
  2. What are the downstream results of the single-objective variants, and how were the baselines trained and adapted to three modalities?
  3. Which modalities and patch projections are used for each downstream dataset, and are modalities ever combined?
  4. Is there evidence that the conclusions hold beyond three image modalities and frozen evaluation?
5 comments

W1. Evidence for the intended interactions

The reviewer's observation is on-point, and we agree that further evidence would help establish how the objectives encourage the intended interactions. We provide additional results below.

Controlled experiment with known information structure.

To examine whether the objectives encourage their intended interactions, we constructed a controlled dataset where each sample consists of three inputs, aa, bb, and cc, each represented by a sequence of tokens. Each input contains the same shared variable ss, its own unique variable uiu_i, and a synergistic variable xix_i (i∈{a,b,c}i \in \{a,b,c\}). We independently sample ss, uau_a, ubu_b, ucu_c, xax_a, and xbx_b uniformly over four classes and define xc=−(xa+xb) mod 4x_c = -(x_a + x_b) \bmod 4 (i.e., the non-negative remainder after division by 4). By construction, ss is shared across all three inputs, each uiu_i is unique to its corresponding input, and predicting xcx_c requires combining xax_a and xbx_b, as neither variable alone provides information about xcx_c.

We pretrained Tri-JEPA and its single-objective variants on 50,000 samples using the same training protocol and evaluated them on 3,000 held-out samples. We used linear probes on frozen encoder representations to assess shared and unique information. To assess synergy, we evaluated the prediction of xcx_c using the two inputs jointly. The results below provide further evidence that each objective encourages its intended interaction while Tri-JEPA emphasizes all three.

Table 1. Prediction accuracy for shared, unique, and synergistic variables on controlled data.

ModelSharedUniqueSynergy
Chance0.2500.2500.250
Shared only0.9990.3270.251
Unique only0.3510.9990.248
Synergistic only0.2960.2521.000
Tri-JEPA0.9940.9400.999

Q1. MI estimation and reliability.

To assess whether our MI findings are robust to the choice of estimator, we evaluate cross-sensor MI using three estimators: Gaussian MI [1], KSG [2], and latent MI [3]. We also include shuffled-pair controls. As shown below, all three estimators consistently rank Tri-JEPA highest. To further assess reproducibility, we independently pretrained a second Tri-JEPA model, which achieves comparable Gaussian MI (8.505 ± 0.056 bits).

Table 2. Cross-sensor MI estimates (bits) averaged across three sensor pairs using 10,000 held-out EO tiles. Mean ± standard deviation over five seeded data resamples and estimator runs.

ModelGaussian MIKSGLatent MI
Shuffled0.000 ± 0.0010.000 ± 0.0070.001 ± 0.009
Shared only7.533 ± 0.0403.329 ± 0.0173.856 ± 0.028
Unique only2.151 ± 0.0211.286 ± 0.0112.071 ± 0.014
Synergistic only6.997 ± 0.0482.944 ± 0.0223.946 ± 0.055
Tri-JEPA8.623 ± 0.0743.421 ± 0.0254.280 ± 0.041

Q1. Task-dependent contributions.

Thank you for kindly pointing this out. We have corrected Sec. 5.2. Our overall takeaway remains unchanged that individual objectives contribute differently across tasks and that Tri-JEPA achieves the best performance.

References

[1] Cover and Thomas (2006). Elements of Information Theory. 2nd Edition. Wiley.

[2] Kraskov et al. (2004). Estimating Mutual Information. Physical Review E.

[3] Gowri et al. (2024). Approximating Mutual Information of High-Dimensional Variables Using Learned Representations. NeurIPS.

W2 & Q2. Sources of improvement and baseline settings

We thank the reviewer for the helpful pointers to examine the sources of improvement.

Single-objective variants.

To assess the contribution of each objective, we pretrained shared-only, unique-only, and synergistic-only Tri-JEPA variants under the same protocol. The table below further supports the benefit of combining the three objectives.

Table 3. Single-objective ablations.

ModelClassificationSegmentationObject DetectionChange Detection
Shared only0.7400.6400.8520.716
Unique only0.7040.6110.8590.709
Synergistic only0.7360.6490.8530.721
Tri-JEPA0.7570.6620.8690.739

Baseline training and adaptation.

All methods follow the same pretraining and downstream protocols using a shared ViT-B encoder and modality-specific patch embeddings. We used the official implementations of FactorCL, CoMM, and InfMasking, preserving their fusion modules and objectives. CoMM and InfMasking natively support multiple modalities. We adapted FactorCL by averaging its loss across the three sensor pairs. MultiMAE and ImageBind retain their multimodal objectives. DINO uses self-distillation between augmented views and I-JEPA uses within-modality masked prediction with an EMA target encoder.

Compute efficiency.

We thank the reviewer for this question. Under identical settings, Tri-JEPA requires substantially less computation per step than the baselines. FactorCL, CoMM, and InfMasking encode two augmented views of each sample. CoMM and InfMasking additionally run a fusion transformer for every modality subset, and InfMasking further processes five masked copies of each subset. In contrast, Tri-JEPA encodes the visible tokens from a single view and reuses them across all three objectives. Each objective simply passes a different combination of the encoded tokens to a lightweight predictor. This avoids repeated encoding and substantially reduces computation.

Table 4. Pretraining computation under identical settings.

ModelTFLOPs / stepRelative compute
FactorCL56.03.6×
CoMM62.44.1×
InfMasking186.212.1×
Tri-JEPA15.41.0×

Data efficiency.

We evaluated data efficiency using subsets of pretraining data. With only half of the data, Tri-JEPA outperforms many of the baselines pretrained on the full dataset.

Table 5. Tri-JEPA pretrained on subsets.

Pretraining dataClassificationSegmentation
25%0.7310.625
50%0.7490.642
100%0.7570.662

W3 & Q3. Downstream protocol, modality fusion, and per-dataset results

Thank you for the comments. We clarify the downstream protocol with additional results below.

Modalities and patch projections.

Following the GEO-Bench evaluation framework, we use consistent downstream protocols across baselines. We map m-eurosat, m-bigearthnet, m-brick-kiln, m-so2sat, m-SA-crop-type, m-cashew-plantation, and FoTW to the pretrained S2 projection. We use the S1 projection for DSSDD and BraDD-S1TS, and the RGB projection for m-forestnet, m-pv4ger, m-pv4ger-seg, m-nz-cattle, and LEVIR-CD.

Downstream modality fusion.

Table 1 uses a single mapped sensor. To evaluate jointly observed modalities, we use m-so2sat, which provides S1 and S2 inputs. Each sensor is processed through its corresponding patch embedding and the representations are concatenated for downstream linear probing. Tri-JEPA outperforms the single-objective synergistic variant, and removing the synergistic objective substantially reduces accuracy.

Table 6. Fused-sensor evaluation on m-so2sat via linear probing on fused features.

Modelm-so2sat
Synergistic only0.548
w/o synergistic0.538
Tri-JEPA0.582

Per-dataset results.

Kindly find the per-dataset results in the appendix. Tri-JEPA achieves the best average rank of 1.07 across all datasets, demonstrating its consistent benefits.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Summary

We thank all reviewers for their accurate summaries and thoughtful feedback on our work. We are encouraged that the reviewers recognized the simplicity and effectiveness of Tri-JEPA.

In response to the valuable feedback across reviewers, we have made several improvements to the manuscript, now updated. At a high level, we summarize the main clarifications and additions we have incorporated:

  • Reviewer vLAg suggested strengthening the evidence for the intended interactions and providing further details on the experimental protocols. We address these points by introducing a controlled experiment with known information structure, extending our mutual information analysis with multiple estimators and reliability checks, and reporting single-objective ablations. We also clarify the pretraining and downstream protocols and provide additional results on modality fusion and per-dataset performance.

  • Reviewer v5D2 suggested a comparison with a merged-token JEPA baseline to better understand the benefits of separating the prediction objectives. We incorporate this comparison and show that Tri-JEPA consistently improves performance under both InfoNCE and smooth L1 losses. Our attention analysis further illustrates how separating prediction contexts encourages cross-modal learning. We also extend our evaluation across backbone sizes and pretraining data scales.

  • Reviewer 77wg encouraged us to further distinguish Tri-JEPA from existing multimodal learning approaches and strengthen the evidence for its intended interactions, efficiency, and generalizability. We clarify our contribution of formulating shared, unique, and synergistic interactions as distinct predictive objectives within a JEPA-based framework and provide controlled experiments supporting this design. We also demonstrate substantial compute and data efficiency gains and extend our evaluation to heterogeneous modalities, including text and audio.

We thank all reviewers once again for their valuable feedback, which has helped us strengthen the motivation, empirical evidence, and broader applicability of Tri-JEPA.

Area_Chair_Z8NjOpenReview ↗

Discussion reminder

Dear Reviewers,thank you for your time and thoughtful evaluation of the Paper. The authors have now submitted their rebuttal addressing the questions and concerns raised in your initial reviews. Please carefully review the authors' responses alongside the other reviews, and participate in the discussion to share your updated thoughts and comments. To help us reach a well-informed meta-review decision, please let us know if the rebuttal adequately resolves your concerns and whether you plan to adjust your final rating. Please submit your post-rebuttal comments and updated evaluation on the review portal. Thank you again for your valuable service and contribution to the review process.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.