acceptodds
Under review as a conference paper at ICLR 2027

K-JEPA: From Point to Distributional Prediction by Kernelized Joint-Embedding Predictive Architectures

Abstract

Joint-Embedding Predictive Architectures (JEPA) learn representations by predicting latent target features from a visible context. Although effective, standard JEPA objectives treat prediction as a point-estimation problem: the population-optimal predictor recovers a single point summary of the conditional target distribution. When a visible context admits multiple plausible masked or future targets, this point estimate retains insufficient distributional information to identify the underlying conditional target distribution. We propose K-JEPA, a novel, simple, kernelized extension that applies a characteristic-kernel transformation to JEPA latent targets while preserving the architecture and prediction setup. Rather than predicting only the latent target , K-JEPA predicts its kernel representation . We prove this changes the population target from a point summary of the latent target distribution to its conditional kernel mean embedding, which uniquely identifies the corresponding conditional target distribution when the kernel is characteristic. Empirically, we evaluate this distributional behavior in controlled settings and introduce a systematic target-retrieval comparison for direct evaluation of JEPA target prediction across heterogeneous formulations and modalities. We complement this evaluation with probabilistic scoring, calibration, selective prediction, prediction-set efficiency, and downstream transfer analyses. Our results characterize when K-JEPA improves target prediction and how these changes relate to downstream performance and reliability. Code is made available to the reviewers during the discussion process.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
JEPA, Kernel Methods, Predictive Reliability, Self-Supervised Learning, Responsible AI

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_YTpyOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes K-JEPA, a kernelized extension of Joint-Embedding Predictive Architectures (JEPA) that aims to move beyond point-valued latent prediction toward distribution-aware prediction. The method introduces an auxiliary kernel-feature prediction objective based on characteristic kernel mean embeddings, implemented using random Fourier features while retaining the original JEPA objective. Theoretical analysis shows that the conditional kernel mean embedding uniquely identifies the target distribution under a characteristic kernel at the population level. Experiments across image, audio, tabular, and world-model settings evaluate the method in terms of downstream performance, target retrieval, uncertainty calibration, and predictive reliability. The results demonstrate improvements across multiple settings, although the gains vary across models and tasks.

Critical strengths
  1. The paper addresses an interesting limitation of conventional JEPA objectives by extending point-valued latent prediction toward distribution-aware prediction. The proposed kernelized formulation is simple and can be incorporated into different JEPA architectures.

  2. The experimental evaluation covers image, audio, tabular, and world-model settings, demonstrating the broad applicability of the proposed framework. Improvements in target retrieval, calibration, and selective prediction provide useful evidence beyond conventional downstream performance.

Critical weaknesses

1. Limited methodological and theoretical novelty. The central theoretical results largely follow from established properties of conditional mean regression and characteristic kernel mean embeddings. The practical method mainly introduces an additional kernel-feature prediction loss into existing JEPA architectures. The authors should better clarify the novelty beyond applying established conditional kernel embedding techniques to JEPA.

2. Insufficient ablation studies. The paper lacks comprehensive ablations to isolate the contribution of the proposed distributional prediction mechanism. In particular, comparisons with parameter-matched auxiliary prediction heads, alternative nonlinear target transformations, and different kernel-feature dimensions are needed. Without these experiments, it is unclear whether the improvements arise specifically from distributional prediction or additional representation regularization.

3. Modest and inconsistent downstream improvements. Although K-JEPA improves several predictive and reliability metrics, its downstream classification gains are relatively small in many settings. For example, the improvements on ImageNet-100 are only 1.1 and 1.0 percentage points for I-JEPA and LpJEPA, respectively, while LeJEPA slightly decreases from 88.0% to 87.9%. The authors should provide more comprehensive statistical evidence to establish the consistency and significance of these gains.

4. Gap between distributional theory and empirical validation. The distribution-identification guarantee applies to the population characteristic kernel, whereas the practical implementation uses finite random features. Moreover, improvements in retrieval and downstream classification do not directly establish accurate recovery of multimodal conditional target distributions. More direct distributional evaluations and sensitivity analyses are needed to support the main theoretical motivation.

5. Limited evaluation of full pretraining and computational efficiency. The image and audio experiments primarily involve short continuation training from pretrained checkpoints rather than full self-supervised pretraining. Consequently, the benefits of K-JEPA in standard end-to-end JEPA pretraining remain insufficiently demonstrated. In addition, training-time and memory overhead should be reported to substantiate the claimed efficiency of the method.

6. Unclear relationship between target prediction and downstream performance. Some substantial improvements in target retrieval lead to only modest downstream gains. For example, Audio-JEPA's R@1 on SpeechCommands increases from 14.1% to 95.3%, while downstream accuracy improves by only 2.2 percentage points.

Questions for authors

Please see the Weaknesses section above.

8 comments

We thank the reviewer for their helpful comments.

YTpy.q1. [...] theoretical results largely follow from established properties of conditional mean regression and characteristic kernel mean embeddings. The practical method mainly introduces an additional kernel-feature prediction loss into existing JEPA architectures. The authors should better clarify the novelty beyond applying established [...] techniques to JEPA.

YTpy.a1: We clarify that our contribution is not the introduction of conditional mean embeddings or characteristic kernels themselves, which are established tools [5]. K-JEPA's nontrivial methodological novelty lies in integrating distribution-identifying prediction directly into JEPA's shared representation learning, which makes a conditional kernel mean embedding the auxiliary predictor's population-optimal target (Corollary 1).

Unlike existing neural-CME methods [3], K-JEPA incorporates this principle directly into self-supervised representation learning and improves the learned representations while keeping the original JEPA objective and architecture (Eq. 11). It also differs from KerJEPA [4], which uses kernel discrepancies to regularize representations, and from variational, diffusion-based, or branching JEPA extensions (Table 1) which require additional assumptions or prediction mechanisms.

Our theoretical analysis further establishes how adaptive balancing and finite random features affect the distributional guarantees (Propositions 3-4). Beyond the method itself, we introduce a common evaluation of JEPA's native target prediction and reliability across various architectures, alongside downstream evaluation (Tables 2–8). Our contributions concern both a practical way to extend JEPA from point to distributional prediction and a systematic evaluation of its effects.

YTpy.q2. The paper lacks comprehensive ablations to isolate the contribution of the proposed distributional prediction mechanism. In particular, comparisons with parameter-matched auxiliary prediction heads, alternative nonlinear target transformations, and different kernel-feature dimensions are needed.

YTpy.a2: To isolate the contribution of the proposed distribution-sensitive prediction mechanism, we additionally evaluated various target-prediction controls under matched training and across five JEPA hosts:

Table A. Comparison of native JEPA, auxiliary MSE, random linear, random nonlinear, and K-JEPA objectives using downstream performance, calibration, and selective-prediction metrics. World-model results use SM-MNIST, while I-JEPA uses ImageNet-100. All results follow the corresponding evaluations described in the manuscript.

HostMethodAcc ↑Macro-F1 ↑AUPRC ↑Brier ↓NLL ↓ECE ↓Err-AUROC ↑AURC ↓
V-JEPANative75.043.126.50.3700.5601.751.424.3
Aux. MSE75.143.728.40.3730.5601.653.123.3
Random linear75.043.426.00.3740.5611.651.324.8
Random nonlinear75.143.125.40.3750.5621.750.624.6
K-JEPA74.951.941.00.3400.5201.470.014.0
C-JEPANative75.042.926.30.3700.5600.551.424.4
Aux. MSE75.042.926.20.3750.5621.552.223.7
Random linear75.042.927.70.3740.5611.654.322.3
Random nonlinear75.042.928.30.3730.5601.554.422.4
K-JEPA75.042.928.80.3700.5600.755.621.8
VJENative75.042.924.90.3700.5601.550.025.4
Aux. MSE75.043.332.20.3680.5530.958.520.7
Random linear75.043.330.20.3700.5542.559.118.9
Random nonlinear75.042.924.90.3750.5621.550.025.4
K-JEPA75.043.632.90.3700.5501.660.818.9
I-JEPANative62.261.867.20.5101.4003.983.615.1
Aux. MSE62.261.966.90.5031.4033.483.315.2
Random linear61.561.267.10.5041.4124.084.315.2
Random nonlinear61.561.266.60.5071.4304.184.015.4
K-JEPA63.362.067.00.4901.4103.583.315.2
D-JEPANative76.357.146.70.3300.5001.170.912.9
Aux. MSE76.658.448.40.3220.4911.371.712.3
Random linear76.859.748.60.3220.4911.271.212.4
Random nonlinear76.358.247.70.3240.4931.371.812.5
K-JEPA76.458.848.30.3200.4901.472.212.3

Across JEPA hosts, K-JEPA achieves the highest downstream Macro-F1 on four and the highest AUPRC on three, with improvements in probabilistic prediction and selective reliability. The strongest differences appear for V-JEPA, where K-JEPA improves Macro-F1 to 51.9, AUPRC to 41.0, Error AUROC to 70.0, and AURC to 14.0, clearly outperforming all auxiliary objectives. The relative advantages vary across hosts and metrics, but the complete evaluation shows that generic supervision does not consistently reproduce K-JEPA's strength.

We provide further ablations in our next comment to the reviewer.

YTpy.a2 continuation: We additionally evaluated random-feature dimension sensitivity on D∈{64,256,1024}D\in\{64,256,1024\} across three SM-MNIST hosts in comparable settings:

Table B. This table shows the random-feature dimension sensitivity check on SM-MNIST. Safety averages Error-AUROC and (1 − AURC), while State error averages normalized MAE and RMSE. All results follow the training and evaluation described in the manuscript.

HostDMacro-F1 ↑AUPRC ↑Safety ↑State error ↓R@10 ↑MRR ↑NLL ↓
B-JEPA+K-JEPA6443.225.362.50.447.34.10.56
25643.527.665.60.4410.76.20.56
102443.027.766.40.448.34.30.56
V-JEPA+K-JEPA6442.925.362.50.447.04.00.56
25651.941.078.00.4110.96.00.52
102442.925.162.60.447.24.10.56
VJE+K-JEPA6443.228.068.20.439.35.10.57
25643.632.971.00.4410.75.50.55
102443.533.270.40.429.15.70.57

The standard D=256D=256 configuration shows strongest overall performance and distribution quality, and performs particularly well for V-JEPA. B-JEPA exhibits smaller variations, while VJE shows metric-dependent sensitivity, with D=256D=256 achieving better NLL, selective safety, and target prediction, and D=1024D=1024 achieving slightly better AUPRC, state error, and MRR.

Finally, we investigated fixed versus adaptive loss weighting. Adaptive gradient balancing is motivated by the different gradient magnitudes of native JEPA and kernel-prediction losses (Section 3.2). To assess fixed weighting, we trained K-JEPA with a constant coefficient λk=0.01\lambda_k=0.01 and compared checkpoints at epochs 10 and 20.

Table C. Fixed-weight training sensitivity on SM-MNIST. The results show within-run comparison of checkpoints at epochs 10 and 20 under a fixed kernel-loss coefficient λk=0.01\lambda_k=0.01. Safety averages Error-AUROC and (1 − AURC).

HostEpochAcc ↑Macro-F1 ↑AUPRC ↑NLL ↓Safety ↑State error ↓
B-JEPA+ Fixed-weight K-JEPA1075.5656.2343.940.499779.730.4023
2075.3453.3442.340.509478.460.4077
C-JEPA+ Fixed-weight K-JEPA1075.0542.8727.600.561066.220.4268
2075.0542.8726.870.561665.040.4296
V-JEPA+ Fixed-weight K-JEPA1074.9652.3241.070.514678.320.4060
2074.9044.7433.810.544572.540.4239

The fixed-weight experiments show deterioration during continued training, with the magnitude depending on the host. The clearest pattern appears for V-JEPA, where Macro-F1 decreases from 52.32 to 44.74, AUPRC from 41.07 to 33.81, and selective safety from 78.32 to 72.54, alongside worsening NLL and state error. These results support the importance of adaptive loss balancing rather than using a fixed coefficient across training.

YTpy.q3. Modest and inconsistent downstream improvements [...] the improvements on ImageNet-100 are only 1.1 and 1.0 percentage points [...] LeJEPA slightly decreases from 88.0% to 87.9% [...] more comprehensive statistical evidence [is needed].

YTpy.a3: We clarify that K-JEPA's central contribution is distribution-aware latent prediction, not simply maximizing downstream classification accuracy. On ImageNet-100, all three image methods improve target R@1, MRR, and NLL, with further improvements in Brier and selective AURC for LpJEPA and LeJEPA, while I-JEPA remains at already low values (Table 4). Importantly, these gains also extend to external zero-shot retrieval on Copydays, where all nine model–missingness settings improve R@1, MRR, NLL, and ECE, with Brier improving or remaining unchanged (Table 5). These predictive and reliability benefits are achieved while preserving or improving downstream representation quality. All three image methods improve downstream Brier and ECE, accuracy improves for I-JEPA and LpJEPA, and LeJEPA's accuracy changes by only 0.1 points while improving AUPRC and Error-AUROC (Table 3). Thus, the image results demonstrate consistent benefits across target prediction, probabilistic scoring, calibration, and selective reliability, which cannot be captured by classification accuracy alone. This broader pattern also extends across audio, tabular, and world-model experiments (Tables 2, 6–8; Appendix Tables 11–12).

In addition, the improvements extend consistently across other modalities and tasks. K-JEPA improves accuracy, macro-F1, and AUPRC across all six audio model-dataset comparisons (Table 2), and downstream accuracy on all three tabular datasets (Table 6). Tables 11–13 also report various statistically significant gains across multiple tasks and JEPA formulations. Our results show consistent benefits across multiple performance and reliability metrics, even when individual accuracy gains are modest.

We report the paired ImageNet-100 results in the next comment.

YTpy.a3 continuation: To further address statistical uncertainty, we conducted paired ImageNet-100 evaluations on 5,000 matched examples using 1,000 bootstrap resamples. The following table (Table D) reports downstream differences between native JEPA and K-JEPA, along with estimated standard errors.

Table D. Differences (Δ) are K-JEPA minus native JEPA, calculated from the published manuscript Table 3 results. Standard errors (SEs) were estimated using 5,000 matched evaluation examples and 1,000 paired bootstrap resamples. Accuracy SEs were derived from paired correctness counts, while the remaining SEs were approximated from paired-bootstrap 95% confidence-interval widths. An asterisk (*) shows a favorable K-JEPA improvement over native JEPA exceeding one estimated standard error (SE).

MetricK-JEPA vs. I-JEPASEK-JEPA vs. LpJEPASEK-JEPA vs. LeJEPASE
Accuracy ↑+1.1*0.46+1.0*0.56-0.10.29
Macro-F1 ↑+0.20.44+0.9*0.59-0.10.28
AUPRC ↑-0.20.32+0.6*0.42+0.6*0.13
Brier ↓-0.02*0.004-0.01*0.005-0.01*0.002
NLL ↓+0.010.009-0.02*0.0130.000.004
ECE ↓-0.330.49-0.500.57-0.090.33

Across these results, K-JEPA improves 13 of 18 metrics, including all six for Rectified LpJEPA. Our additional paired evaluations corroborate this favorable pattern, with several improvements exceeding one estimated standard error. Together with the consistent target-prediction, calibration, and reliability improvements in Tables 4–5, these findings reinforce that K-JEPA's contribution extends beyond isolated downstream accuracy gains.

YTpy.q4. The distribution-identification guarantee applies to the population characteristic kernel, whereas the practical implementation uses finite random features [...] improvements in retrieval and downstream classification do not directly establish accurate recovery of multimodal conditional target distributions [...] more direct distributional evaluations and sensitivity analyses are needed.

YTpy.A4: We clarify that the finite-feature approximation is already analyzed in Proposition 4 (Appendix A.1.2), which bounds the absolute difference between the squared finite-feature and characteristic-kernel MMDs by 4ηD4\eta_D.d. Furthermore, our evaluations are not limited to retrieval and classification accuracy. Tables 4, 5, 6, and 8 also evaluate Brier and NLL (probabilistic prediction quality), ECE (calibration), and AURC and selective risk (uncertainty reliability), alongside conformal coverage and prediction-set size in Table 8. Our results provide complementary evidence of predictive quality and reliability.

To directly assess conditional-distribution recovery, we additionally evaluated empirical conditional mean embedding (CME) discrepancy on SM-MNIST. For each of 256 contexts, we sampled 32 possible futures and estimated the empirical conditional kernel mean. The learned K-JEPA kernel predictor achieves substantially lower squared discrepancy than the transformed point-prediction reference: 30.52 versus 4.26 for B-JEPA (86.0% reduction) and 169.99 versus 7.77 for V-JEPA (95.4% reduction). The kernel predictor additionally achieves lower discrepancy across all 256 evaluated contexts for both hosts, which provides direct evidence of improved empirical conditional kernel-mean alignment.

We provide further ablations in our next comment.

Reviewer_7gd9OpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes K-JEPA, a plug-in extension to JEPA that adds an auxiliary head to regress onto random Fourier features of the target embedding. The motivation is that squared-loss JEPA regression recovers only the conditional mean, which may fail to capture multimodal target distributions, whereas predicting characteristic-kernel features can recover the conditional kernel mean embedding, which identifies the conditional target distribution at the population level. The authors apply K-JEPA to 13 host methods across image, audio, tabular, and world model settings, evaluating downstream performance, target retrieval, and predictive reliability.

Critical strengths

Strengths

  1. Simple and modular design. K-JEPA adds only an auxiliary prediction head and loss term, without requiring sampling, a decoder, or multiple prediction branches, and is designed to preserve the host configuration.
  2. Clear and principled motivation. Conditional kernel mean embeddings provide a natural framework for addressing the limitations of squared-loss prediction when targets are multimodal.
  3. Broad empirical scope. The evaluation spans four modalities and 13 host methods, including methods that already support distributional prediction. The target-retrieval and reliability evaluations also complement conventional downstream accuracy metrics.
Critical weaknesses
  1. The connection between the theoretical result and the empirical gains is insufficiently established. The propositions characterize the population optimum of the auxiliary head gkg_k, but its output is not directly used in downstream evaluation. The paper does not establish why or under what conditions optimizing this auxiliary objective improves the shared encoder representation on which the downstream results depend. Thus, the theoretical distribution-identification result does not by itself explain the reported empirical gains. Without ablations against non-characteristic features or a matched auxiliary MSE head, the gains also cannot be attributed to the characteristic kernel specifically.
  2. The target-retrieval protocol requires clarification, and some baseline results raise concerns about the evaluation. The paper does not sufficiently clarify which prediction is used for retrieval (e.g., gzg_z, gkg_k, or the concatenated predictor GG), how retrieval candidates are constructed, or how predictive probabilities are obtained for metrics such as Brier score, NLL, and ECE. Some reported baseline R@1 values are notably low, including 1.6 for PupuJEPA and 14.1 for Audio-JEPA. In particular, the increase in Audio-JEPA R@1 from 14.1 to 95.3 warrants careful verification. The adaptive weighting (λk=sJEPA/(sϕ+ε)\lambda_k = s_{\mathrm{JEPA}}/(s_\phi+\varepsilon) with ρ=0.01\rho = 0.01) implies that the auxiliary loss contributes only about 1% of the JEPA gradient norm at ctc_t, applied over a three-epoch adaptation budget. The paper should provide sufficient protocol and implementation details to establish that these comparisons are valid and reproducible.
  3. The experiments do not provide convincing direct evidence of distribution recovery. In the controlled SM-MNIST setting (Table 8), NLL is close to log⁡128\log 128 and the Brier score is around 0.99 for all methods, while Size95 ranges from 119 to 127. If the candidate pool contains about 128 targets, this indicates near-uniform predictions and prediction sets covering nearly all candidates. Several downstream probes in Table 7 (e.g., the C-JEPA, B-JEPA, and VJE baselines, with accuracy 75.0 and Macro-F1 42.9) also appear consistent with majority-class-dominated predictions. These results do not provide convincing evidence that K-JEPA successfully recovers the target distribution in this controlled setting. Moreover, although Appendix B.1 discusses empirical-CME discrepancy, the paper does not report this quantity as a direct diagnostic of the proposed method’s distributional predictions.
Questions for authors
  1. Retrieval and evaluation protocol. Could the authors clarify the target-retrieval setup, including which prediction is used (gzg_z, gkg_k, or GG), how candidates are constructed (and the candidate pool size in the controlled SM-MNIST evaluation), and how predictive probabilities are derived for Brier, NLL, and ECE? It would also help to confirm whether the baseline predictors were trained under the same budget as K-JEPA.
  2. Direct evaluation of distribution recovery. Can the authors report the empirical-CME discrepancy discussed in Appendix B.1, or provide another direct distributional metric, such as MMD to the ground-truth distribution, on a controlled synthetic multimodal task? Such an experiment would help distinguish genuine distributional prediction improvements from gains attributable to auxiliary regularization or changes in the learned representation.
  3. Ablations and theoretical conditions. Can the authors provide ablations using (i) non-characteristic random projections, (ii) a second MSE prediction head with a matched gradient scale, and (iii) a fixed λk\lambda_k? Please also report the realized ηD\eta_D and clarify whether the condition required by Proposition 4 holds in the actual experiments.
5 comments

We thank the reviewer for their helpful comments.

7gd9.q1: The connection between the theoretical result and the empirical gains is insufficiently established. [...] The paper does not establish why or under what conditions optimizing this auxiliary objective improves the shared encoder representation on which the downstream results depend. [...] Without ablations against non-characteristic features or a matched auxiliary MSE head, the gains also cannot be attributed to the characteristic kernel specifically.

7gd9.a1: We clarify that K-JEPA's kernel loss directly trains the shared context encoder (Eq. 11), while downstream evaluation uses the learned encoder representations rather than the auxiliary predictor. This allows us to evaluate whether the additional objective improves representation quality itself.

In addition, we would like to further emphasize the connection between the theoretical results and the empirical gains as follows:

1. Theoretical justification: Prior work provides relevant theoretical foundations connecting self-supervised prediction to downstream representation learning (Lee et al. [2]) and characteristic kernel embeddings to the distinguishability of probability distributions (Fukumizu et al. [1]). Building on these results, we can consider two contexts with identical expected targets but different target distributions.

Standard JEPA's squared-error objective, as well as an additional MSE or linear prediction head, gives identical population-optimal predictions for both contexts and thus cannot distinguish their conditional target distributions through their means alone. In contrast, collapsing these contexts into the same representation introduces an additional population kernel-prediction risk:

= p(1-p)\lVert\mu_{P_a}-\mu_{P_b}\rVert_{\mathcal{H}}^2 = p(1-p)\operatorname{MMD}_k^2(P_a,P_b) > 0,$$ where $p\in(0,1)$ denotes the relative probability of one context and $\mu_P$ its kernel mean embedding. This result holds for fixed target representations and an exact characteristic kernel. Thus, K-JEPA encourages the shared encoder to preserve predictive information that point regression can miss. We have clarified this connection further in the revised paper. **2. Empirical justification**: K-JEPA's kernelized targets provide distribution-sensitive supervision beyond conditional means, with benefits in target prediction, downstream performance, uncertainty calibration, selective safety, conformal reliability, and fairness. These benefits extend across modalities, with K-JEPA improving retrieval, NLL, and ECE across all nine Copydays settings (Table 5), and improving AURC in all six audio downstream comparisons and NLL in five, with one tie at reported precision (Table 2). On stochastic-future prediction, V-JEPA improves macro-F1 from 43.1 to 51.9, AUPRC from 26.5 to 41.0, and AURC from 24.3 to 14.0 (Table 7), while V-JEPA and C-JEPA both achieve conformal coverage closer to 95% with smaller prediction sets (Table 8). Furthermore, K-JEPA improves 9 of 12 demographic fairness comparisons on MIMIC-IV (Appendix Table 10). We also clarify that target retrieval and downstream classification measure different aspects of representation quality, and improvements in one do not necessarily translate proportionally to the other. This is also observed in the multimodal learning literature [4,5]. For example, SaCo Loss [4] (CVPR 2024) improves Flickr30K retrieval R@1 by 7.9 percentage points while improving ImageNet classification by only 1.0 point. Similarly, CAV-MAE Sync [5] (CVPR 2025) reports a 10.4-point retrieval improvement between CAV-MAE variants, while AudioSet classification decreased by 3.9 points. In addition, K-JEPA's effect depends on the original JEPA formulation. Audio-JEPA uses point-valued latent prediction, whereas methods such as D-JEPA and B-JEPA already incorporate distributional prediction mechanisms. Consequently, the additional information introduced by K-JEPA can vary across methods, leading to different magnitudes of improvement. Our results (Tables 2–8 and Figure 1(c)) demonstrate benefits at both the target and downstream levels, without requiring these improvements to be proportional. We report various target-prediction ablations in our next comment to the reviewer.

7gd9.a1 continuation: To address the reviewer's concern about whether the empirical gains arise from the characteristic-kernel objective rather than generic auxiliary supervision, we additionally evaluated multiple target-prediction controls under the same training protocol across five JEPA hosts. The following table reports the complete comparison of their results.

Table A. Comparison of native JEPA, auxiliary MSE, random linear, random nonlinear, and K-JEPA using downstream performance, probabilistic scoring, calibration, and selective reliability metrics.

HostMethodAcc ↑Macro-F1 ↑AUPRC ↑Brier ↓NLL ↓ECE ↓Err-AUROC ↑AURC ↓
V-JEPANative75.043.126.50.3700.5601.751.424.3
Aux. MSE75.143.728.40.3730.5601.653.123.3
Random linear75.043.426.00.3740.5611.651.324.8
Random nonlinear75.143.125.40.3750.5621.750.624.6
K-JEPA74.951.941.00.3400.5201.470.014.0
C-JEPANative75.042.926.30.3700.5600.551.424.4
Aux. MSE75.042.926.20.3750.5621.552.223.7
Random linear75.042.927.70.3740.5611.654.322.3
Random nonlinear75.042.928.30.3730.5601.554.422.4
K-JEPA75.042.928.80.3700.5600.755.621.8
VJENative75.042.924.90.3700.5601.550.025.4
Aux. MSE75.043.332.20.3680.5530.958.520.7
Random linear75.043.330.20.3700.5542.559.118.9
Random nonlinear75.042.924.90.3750.5621.550.025.4
K-JEPA75.043.632.90.3700.5501.660.818.9
I-JEPANative62.261.867.20.5101.4003.983.615.1
Aux. MSE62.261.966.90.5031.4033.483.315.2
Random linear61.561.267.10.5041.4124.084.315.2
Random nonlinear61.561.266.60.5071.4304.184.015.4
K-JEPA63.362.067.00.4901.4103.583.315.2
D-JEPANative76.357.146.70.3300.5001.170.912.9
Aux. MSE76.658.448.40.3220.4911.371.712.3
Random linear76.859.748.60.3220.4911.271.212.4
Random nonlinear76.358.247.70.3240.4931.371.812.5
K-JEPA76.458.848.30.3200.4901.472.212.3

Across five JEPA hosts, K-JEPA achieves the highest downstream Macro-F1 on four and the highest AUPRC on three, while providing improvements in probabilistic prediction and selective reliability. The strongest differences appear for V-JEPA, where K-JEPA improves Macro-F1 to 51.9, AUPRC to 41.0, Error AUROC to 70.0, and AURC to 14.0 and clearly outperforms all auxiliary objectives. The advantages vary across hosts and metrics, but the complete evaluation shows clearly that these generic target-prediction methods could not reproduce K-JEPA's combined benefits.

7gd9.q2: The target-retrieval protocol requires clarification [...] In particular, the increase in Audio-JEPA R@1 from 14.1 to 95.3 warrants careful verification [...] The paper should provide sufficient protocol and implementation details to establish that these comparisons are valid and reproducible.

7gd9.A2: We clarify that Table 4 evaluates zero-shot target retrieval on SpeechCommands, with audio models trained on AudioSet. The original JEPA methods use their trained native predictor (g_z), while K-JEPA uses its learned kernel predictor (g_k) to evaluate the kernelized target's quality and distribution. Each query is evaluated against 64 target candidates (one correct target and 63 alternatives from other recordings), ranked by squared distance. Brier, NLL, and ECE are computed from temperature-scaled softmax probabilities, with temperature fitted on 13 calibration queries and evaluated on the remaining 51.

Regarding the low baselines, the evaluation is zero-shot, so performance depends on what each original method learned during pretraining. In fact, the original Audio-JEPA paper [3] explicitly reports poor SpeechCommands transfer and identifies fine-grained speech discrimination as a limitation. WavJEPA instead learns temporally detailed waveform representations, while PupuJEPA was originally designed and pretrained for music. These architectural and pretraining differences explain the substantially different baseline retrieval performance and room for improvement.

Regarding Audio-JEPA's R@1, the improvement is due to the learned kernel predictor's ability to retrieve latent targets, rather than an equally large change in the shared encoder. Importantly, λk\lambda_k is adaptively adjusted during training, and the gradient contribution applies specifically to the shared context representation, not to the kernel predictor's training or predictive capacity. The kernel predictor is directly optimized over 3 epochs during the AudioSet adaptation. We have clarified these distinctions in the paper and will provide the complete evaluation protocol, checkpoint details, and code for reproducibility.

7gd9.q3: The experiments do not provide convincing direct evidence of distribution recovery [...] NLL is close to log 128 and the Brier score is around 0.99 [...] several downstream probes [...] appear consistent with majority-class-dominated predictions [...] the paper does not report [empirical-CME discrepancy].

7gd9.A3: We clarify that K-JEPA's objective is to improve the predictive representations of existing JEPA methods, rather than guarantee perfect predictions independently of the underlying model. Its effectiveness therefore depends on the original method and evaluation setting. For example, PupuJEPA's target R@1 improves from 1.6 to 4.7, whereas Audio-JEPA improves from 14.1 to 95.3 (Table 4). Different absolute performance levels do not contradict K-JEPA's objective of improving the underlying baseline.

We also note that the rounded Brier scores in Table 8 hide K-JEPA's statistically significant improvements shown in appendix (Table 13). The full-precision results show reductions for four of five JEPA methods, including three with p<0.05 and a fourth with p<0.10 under paired permutation tests (Appendix Table 13). For example, B-JEPA's Brier score improves significantly under the paired permutation test (p<0.01p<0.01; Appendix Table 13), with a 95% confidence interval for the improvement of [0.0005, 0.0019]. Thus, the rounded 0.99 scores do not show that K-JEPA fails to improve predictive reliability.

In addition, the majority-class interpretation overlooks the statistically significant improvements reported for the same five model pairs evaluated in Table 8. The cited 75% accuracy and 42.9 macro-F1 concern selected baseline classification results, whereas K-JEPA significantly improves AUPRC for all five methods and macro-F1 for four (p<0.01, Appendix Table 12). K-JEPA also improves all 30 future-state error comparisons across position, velocity, and normalized state (p<0.01, Appendix Table 11). The results show that K-JEPA learns substantially more useful predictive information than the selected accuracy and retrieval-probability figures suggest.

We additionally evaluated empirical conditional kernel-mean discrepancy on stochastic-future prediction, using 256 contexts with 32 sampled futures each. The K-JEPA kernel predictor achieves substantially lower discrepancy than the transformed point-prediction reference for both B-JEPA and V-JEPA (see 7gd9.A5 for results and protocol). This provides direct evidence of improved empirical kernel-mean alignment, which complements the downstream and reliability improvements reported above.

7gd9.q4: Retrieval and evaluation protocol. Could the authors clarify the target-retrieval setup, including which prediction is used (g_z, g_k, or G), how candidates are constructed (and the candidate pool size in the controlled SM-MNIST evaluation), and how predictive probabilities are derived for Brier, NLL, and ECE? It would also help to confirm whether the baseline predictors were trained under the same budget as K-JEPA.

7gd9.a4: Please refer to 7gd9.a2 for the detailed target-retrieval protocol, including predictor selection, candidate construction, and probability calibration. For the controlled SM-MNIST evaluation, retrieval uses 128 candidates per query (one observed correct target and 127 distractors from other sequences). Native JEPA and K-JEPA are trained under the same host-specific training budget and evaluation protocol, as detailed in Appendix B.2.1–B.2.3 and Table 9.

7gd9.q5: Can the authors report the empirical-CME discrepancy discussed in Appendix B.1, or provide another direct distributional metric, such as MMD to the ground-truth distribution, on a controlled synthetic multimodal task?

7gd9.a5: We additionally evaluated empirical conditional mean embedding (CME) discrepancy on Stochastic Moving MNIST, as described in Appendix B.1. For each of 256 contexts, we sampled 32 possible futures and estimated the empirical conditional kernel mean by averaging their random-feature embeddings. We then compared the squared discrepancy of K-JEPA's learned kernel prediction against the transformed point prediction while using a prediction horizon of 10.

Table B. Empirical conditional kernel-mean discrepancy (lower is better).

HostPoint reference ↓K-JEPA kernel predictor ↓Reduction
B-JEPA30.524.2686.0%
V-JEPA169.997.7795.4%

The kernel predictor achieves lower discrepancy for all 256 evaluated contexts on both hosts, which shows better alignment with the empirical conditional kernel mean than the corresponding point-prediction reference. This directly evaluates the kernel-prediction objective, which complements the already observed performance, retrieval, and distributional gains in the paper. In addition, the matched auxiliary-objective comparisons in 7gd9.a1 provide complementary evidence regarding the contribution of kernelized supervision.

Reviewer_TSbUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a kernelized target representation to turn prevalent point-prediction of I-JEPA into distribution prediction, aiming to improve prediction quality. This simple change to the training objective is easily combined with prior JEPA-based approaches. This paper confirms its effectiveness on various kind of downstream tasks, including image classification, audio classification, and calibration. The proposed K-JEPA yields consistent improvements across most of benchmarks retaining original JEPA framework.

Critical strengths
  • The paper provides a simple extension of JEPA. The connection between standard latent regression and kernel-based target prediction is clear, and the proposed method can be added to existing JEPA frameworks with relatively small modifications.
  • The authors evaluate the method across multiple modalities and JEPA variants, and consider both downstream performance and target-prediction/reliability metrics.
Critical weaknesses
  • The motivation assumes that the same context can correspond to multiple plausible targets. However, in standard masked-image JEPA, the target for a given image and mask is fixed. The paper mainly demonstrates the ambiguous-target issue in controlled settings, but it is unclear how often this problem appears in the main real-world experiments. This makes the connection between the motivation and the practical setting less convincing.
  • In practice, K-JEPA adds an auxiliary prediction loss on fixed nonlinear features while keeping the original JEPA loss. Therefore, the gains may also come from the additional nonlinear supervision itself. In L1043-L1045, it is stated that the main image- and audio-based experiments adopt pre-trained models as an initialization, and fine-tune them for several epochs. However, this training setup does not show whether K-JEPA can improve representation quality across full pre-training epochs. To study this, it is encouraged to add a full pre-training run on the presented audio and image datasets using the proposed K-JEPA objective.
Questions for authors
  • While Corollary 1 shows K-JEPA turns point prediction of the original I-JEPA into distribution prediction, it is still unclear such transformation in prediction target yields better downstream performance (or linear separability as in LeJEPA). It is suggested to add theoretical analysis beyond prediction quality of K-JEPA.
  • While various downstream experimental results are already provided, it still lacks further analysis such as feature geometry and design choice of kernels. It should be included ablation experiments for choice of kernel methods and visualization or quantitative evaluation on the learned feature geometry.
  • How sensitive is K-JEPA to the choice of kernel and the random-feature dimension (D)? The current experiments use kernel selection on the validation set and fix (D=256).
  • For the image experiments, why is the evaluation mainly conducted on ImageNet-100 rather than ImageNet-1K?
  • In Table4, combining K-JEPA into Audio-JEPA leads to a substantial improvements in the SpeechCommands dataset, while all other results show different improvement trend. Could you provide some reason about such difference in performance improvements?
  • How established benefit of K-JEPA on prediction quality can relate to representation quality on downstream tasks?
6 comments

We thank the reviewer for their helpful comments.

TSbU.q1: In standard masked-image JEPA, the target for a given image and mask is fixed. [...] it is unclear how often this problem appears in the main real-world experiments.

TSbU.a1: We want to clarify that having a fixed target during training does not mean that a visible context has only one possible target. For instance, in I-JEPA, where the task is to predict the missing part of a masked image in latent space, observing a part of an animal or a tree does not uniquely determine what the hidden regions look like. The animal's fur could have different patterns, while the tree's leaves and branches could have different arrangements. Although each training image provides only one actual completion, many others may be equally plausible given the visible context. Standard JEPA's point-prediction objective reduces this conditional variability to a single latent estimate, which could obscure the uncertainty of missing regions. I-JEPA [1] already acknowledges this ambiguity, albeit to a limited extent. K-JEPA directly addresses it through distribution-sensitive kernel prediction.

Real-world relevance of using distribution targets: Real-world observations often allow multiple plausible completions, even when each training example provides only one realized target. Standard point-prediction objectives cannot fully capture this conditional variability, which is relevant not only to prediction quality but also to uncertainty calibration, robustness to missing information, and fairness across subgroups. K-JEPA directly addresses this limitation through distribution-sensitive kernel supervision, and our experiments across four modalities show clear empirical benefits in predictive reliability, calibration, robustness under missingness, and groupwise fairness (Tables 2–8; Appendix Table 10).

TSbU.q2: The gains may also come from the additional nonlinear supervision itself.

TSbU.a2: We clarify that K-JEPA's additional objective is not simply arbitrary nonlinear supervision. By predicting features of a characteristic kernel, it targets the conditional kernel mean embedding rather than only the conditional mean (Corollary 1). To examine whether generic auxiliary prediction objectives reproduce these benefits, we additionally evaluated multiple target-prediction controls:

Table A. Comparison of native JEPA, auxiliary MSE, random linear, random nonlinear, and K-JEPA objectives using downstream performance, calibration, and selective-prediction metrics. World-model results use SM-MNIST, while I-JEPA uses ImageNet-100. Results follow the evaluations described in the manuscript.

HostMethodAcc ↑Macro-F1 ↑AUPRC ↑Brier ↓NLL ↓ECE ↓Err-AUROC ↑AURC ↓
V-JEPANative75.043.126.50.3700.5601.751.424.3
Aux. MSE75.143.728.40.3730.5601.653.123.3
Random linear75.043.426.00.3740.5611.651.324.8
Random nonlinear75.143.125.40.3750.5621.750.624.6
K-JEPA74.951.941.00.3400.5201.470.014.0
C-JEPANative75.042.926.30.3700.5600.551.424.4
Aux. MSE75.042.926.20.3750.5621.552.223.7
Random linear75.042.927.70.3740.5611.654.322.3
Random nonlinear75.042.928.30.3730.5601.554.422.4
K-JEPA75.042.928.80.3700.5600.755.621.8
VJENative75.042.924.90.3700.5601.550.025.4
Aux. MSE75.043.332.20.3680.5530.958.520.7
Random linear75.043.330.20.3700.5542.559.118.9
Random nonlinear75.042.924.90.3750.5621.550.025.4
K-JEPA75.043.632.90.3700.5501.660.818.9
I-JEPANative62.261.867.20.5101.4003.983.615.1
Aux. MSE62.261.966.90.5031.4033.483.315.2
Random linear61.561.267.10.5041.4124.084.315.2
Random nonlinear61.561.266.60.5071.4304.184.015.4
K-JEPA63.362.067.00.4901.4103.583.315.2
D-JEPANative76.357.146.70.3300.5001.170.912.9
Aux. MSE76.658.448.40.3220.4911.371.712.3
Random linear76.859.748.60.3220.4911.271.212.4
Random nonlinear76.358.247.70.3240.4931.371.812.5
K-JEPA76.458.848.30.3200.4901.472.212.3

Across five JEPA hosts, K-JEPA achieves the highest downstream Macro-F1 on four and the highest AUPRC on three, while providing improvements in probabilistic prediction and selective reliability. The strongest differences appear for V-JEPA, where K-JEPA improves Macro-F1 to 51.9, AUPRC to 41.0, Error AUROC to 70.0, and AURC to 14.0, clearly outperforming all auxiliary objectives. The relative advantages vary across hosts and metrics, but the complete evaluation demonstrates that generic supervision does not consistently reproduce K-JEPA's combined strength.

TSbU.q3: the main image- and audio-based experiments adopt pre-trained models as an initialization [...] this training setup does not show whether K-JEPA can improve representation quality across full pre-training epochs.

TSbU.a3: We want to clarify that our tabular and world-model experiments were trained from scratch (Sect. 4; Tables 6–8), while our image and audio experiments were designed to test whether K-JEPA improves pretrained representations through short adaptation, following the lightweight model adaptation strategy used in relevant literature such as Franchi et al. [5]. To further address the reviewer's concern, we trained native I-JEPA and I-JEPA+K-JEPA from random initialization for 20 epochs on ImageNet-100 under the same training budget. The following table shows pretraining from scratch on image and audio modalities.

Table B. Native JEPA and K-JEPA are trained from random initialization under the same host-specific training budget: 20 epochs on ImageNet-100 for I-JEPA and 15,000 AudioSet training steps for Audio-JEPA, evaluated on SpeechCommands V2 (SCv2) and ESC-50. Performance averages accuracy, macro-F1, and AUPRC; Brier/ECE averages Brier score and ECE; Safety averages Error-AUROC and (1 − AURC).

MetricPerformance ↑Macro-F1 ↑Brier/ECE ↓NLL ↓Safety ↑Coverage gap95 ↓Set size95 ↓
I-JEPA40.339.539.52.3772.12.738.4
+ K-JEPA41.440.638.72.3572.32.537.0
Audio-JEPA (SCv2)62.561.026.71.4482.33.014.7
+ K-JEPA63.461.626.71.3782.73.414.2
Audio-JEPA (ESC-50)71.669.422.21.0485.24.514.4
+ K-JEPA74.071.520.90.9586.94.514.2

At reported precision, K-JEPA improves all seven downstream metrics for I-JEPA, six of seven for ESC-50 with one tie, and five of seven for SpeechCommands. The results show improvements in representation quality under matched pretraining from scratch across both image and audio modalities.

TSbU.q4: It is still unclear if such transformation in prediction target yields better downstream performance [...] suggested to add theoretical analysis beyond the prediction quality of K-JEPA.

TSbU.a4: We follow prior work [2–4] connecting predictive representations to downstream learning. Lee et al. [2] show how self-supervised prediction can yield representations useful for linear prediction; Fukumizu et al. [3] establish a connection between kernel distribution embeddings and classifiability; and Ciliberto et al. [4] relate embedding-prediction error to downstream prediction risk. We have clarified this connection further in the revised paper.

Specifically, two contexts can have the same expected target but different target distributions. Standard JEPA can predict the same representation for both without distinguishing these distributions. K-JEPA instead penalizes this loss of information. At the population level, collapsing these contexts into the same representation introduces an additional kernel-prediction risk:

ΔRk=p(1−p)∥μPa−μPb∥H2=p(1−p)MMD⁡k2(Pa,Pb)>0,\Delta\mathcal{R}_k = p(1-p)\lVert\mu_{P_a}-\mu_{P_b}\rVert_{\mathcal{H}}^2 = p(1-p)\operatorname{MMD}_k^2(P_a,P_b) > 0,

where p∈(0,1)p \in (0,1) is the relative probability of one context. In contrast, the corresponding risk gap under standard squared-error prediction is zero when the conditional means agree. Since the kernel loss also trains the shared context encoder (Eq. 11), K-JEPA encourages representations to preserve these distributional distinctions that point regression can miss. Tables 2–8 further demonstrate the resulting downstream benefits empirically.

References

[1] Assran et al. Self-supervised learning from images with a joint-embedding predictive architecture. CVPR, 2023.

[2] Lee et al. Predicting What You Already Know Helps: Provable Self-Supervised Learning. NeurIPS, 2021.

[3] Fukumizu et al. Kernel Choice and Classifiability for RKHS Embeddings of Probability Distributions. NeurIPS, 2009.

[4] Ciliberto et al. A General Framework for Consistent Structured Prediction with Implicit Loss Embeddings. JMLR, 2020.

[5] Franchi et al. Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models. CVPR, 2024.

TSbU.q5: While various downstream experimental results are already provided, it still lacks further analysis such as feature geometry and design choice of kernels. It should include ablation experiments [...] and visualization or quantitative evaluation on the learned feature geometry.

TSbU.a5: We thank the reviewer for this suggestion. We additionally analyze kernel choice across four characteristic kernel families (RBF, Matérn-1/2, Matérn-3/2, and rational quadratic) on image and world-model architectures. All K-JEPA-specific settings are held fixed (D=256D=256, ρ=0.01\rho=0.01, ℓ=d\ell=\sqrt{d}), with the paper-selected kernels marked by ★. The following tables report downstream performance, target retrieval, and predictive reliability.

Table C. Macro-F1, AUPRC, AURC, R@10, and MRR are percentages; state error is the mean of normalized MAE and RMSE. All results follow the corresponding training and evaluation described in the manuscript.

HostKernelMacro-F1 ↑AUPRC ↑AURC ↓State error ↓R@10 ↑MRR ↑
B-JEPARBF ★43.527.622.60.44010.706.20
Matérn-1/243.626.923.40.4417.834.23
Matérn-3/243.527.123.30.4417.944.20
RQ43.126.524.80.4417.774.28
C-JEPARBF42.927.222.70.4137.774.25
Matérn-1/242.925.724.10.4297.774.27
Matérn-3/242.926.023.80.4247.654.27
RQ ★42.928.821.80.4059.305.10
D-JEPARBF57.247.312.80.3997.904.26
Matérn-1/257.947.412.60.3967.884.28
Matérn-3/258.648.112.20.3957.714.18
RQ ★58.848.312.30.3908.004.60
VJERBF42.924.925.40.4427.684.25
Matérn-1/242.924.925.40.4427.914.24
Matérn-3/242.924.925.40.4427.904.26
RQ ★43.632.918.90.43510.705.50

Table D shows the kernel-family sensitivity on ImageNet-100. Downstream Macro-F1 and AUPRC are reported alongside native target-prediction metrics. All results follow the corresponding training and evaluation described in the manuscript.

HostKernelMacro-F1 ↑AUPRC ↑R@1 ↑MRR ↑NLL ↓AURC ↓
I-JEPARBF61.2366.6699.099.40.050.01
Matérn-1/261.7567.1199.699.70.060.01
Matérn-3/2 ★62.0067.0099.099.30.050.02
RQ61.3666.2098.098.80.080.04
LeJEPARBF87.6193.9593.495.20.812.27
Matérn-1/287.6994.0389.593.11.215.54
Matérn-3/2 ★87.9094.5089.893.50.914.48
RQ88.0194.1186.391.51.276.19

Kernel choice affects how target-representation distances are weighted. RBF favors localized similarities, Matérn kernels vary in smoothness, and RQ captures multiple length scales. These properties interact with the host architecture. For example, C-JEPA's object-centric masking may benefit from RQ's multiscale similarity, which is consistent with its best AUPRC (28.8), AURC (21.8), and state error (0.405). In contrast, B-JEPA's probabilistic product-of-experts formulation shows little kernel sensitivity (Macro-F1: 43.1–43.6). Similarly, LeJEPA's SIGReg encourages isotropic Gaussian representations, where RBF's smooth similarity may favor fine-grained target discrimination. RBF achieves the best retrieval (R@1: 93.4), while downstream Macro-F1 remains nearly constant across kernels (87.61–88.01). Thus, kernel choice can affect target retrieval more strongly than downstream classification, which is also dependent on the host’s learned representation geometry.

For the representative feature-geometry measurements, we calculated effective rank, participation ratio, and pairwise cosine similarity. Both K-JEPA configurations below (Table E) show modest increases in effective rank and participation ratio, alongside lower mean cosine similarity, which shows changes in feature diversity without substantial disruption of the host representations:

Table E. Effective rank, participation ratio, and mean pairwise cosine similarity of the learned encoder representations, comparing native JEPA with K-JEPA.

HostConfigurationEffective rankParticipation ratioMean cosine
I-JEPANative160.3220.320.1137
K-JEPA161.6520.350.1104
LeJEPANative29.584.230.1965
K-JEPA30.664.500.1927

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.