- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes K-JEPA, a kernelized extension of Joint-Embedding Predictive Architectures (JEPA) that aims to move beyond point-valued latent prediction toward distribution-aware prediction. The method introduces an auxiliary kernel-feature prediction objective based on characteristic kernel mean embeddings, implemented using random Fourier features while retaining the original JEPA objective. Theoretical analysis shows that the conditional kernel mean embedding uniquely identifies the target distribution under a characteristic kernel at the population level. Experiments across image, audio, tabular, and world-model settings evaluate the method in terms of downstream performance, target retrieval, uncertainty calibration, and predictive reliability. The results demonstrate improvements across multiple settings, although the gains vary across models and tasks.
Critical strengths
-
The paper addresses an interesting limitation of conventional JEPA objectives by extending point-valued latent prediction toward distribution-aware prediction. The proposed kernelized formulation is simple and can be incorporated into different JEPA architectures.
-
The experimental evaluation covers image, audio, tabular, and world-model settings, demonstrating the broad applicability of the proposed framework. Improvements in target retrieval, calibration, and selective prediction provide useful evidence beyond conventional downstream performance.
Critical weaknesses
1. Limited methodological and theoretical novelty. The central theoretical results largely follow from established properties of conditional mean regression and characteristic kernel mean embeddings. The practical method mainly introduces an additional kernel-feature prediction loss into existing JEPA architectures. The authors should better clarify the novelty beyond applying established conditional kernel embedding techniques to JEPA.
2. Insufficient ablation studies. The paper lacks comprehensive ablations to isolate the contribution of the proposed distributional prediction mechanism. In particular, comparisons with parameter-matched auxiliary prediction heads, alternative nonlinear target transformations, and different kernel-feature dimensions are needed. Without these experiments, it is unclear whether the improvements arise specifically from distributional prediction or additional representation regularization.
3. Modest and inconsistent downstream improvements. Although K-JEPA improves several predictive and reliability metrics, its downstream classification gains are relatively small in many settings. For example, the improvements on ImageNet-100 are only 1.1 and 1.0 percentage points for I-JEPA and LpJEPA, respectively, while LeJEPA slightly decreases from 88.0% to 87.9%. The authors should provide more comprehensive statistical evidence to establish the consistency and significance of these gains.
4. Gap between distributional theory and empirical validation. The distribution-identification guarantee applies to the population characteristic kernel, whereas the practical implementation uses finite random features. Moreover, improvements in retrieval and downstream classification do not directly establish accurate recovery of multimodal conditional target distributions. More direct distributional evaluations and sensitivity analyses are needed to support the main theoretical motivation.
5. Limited evaluation of full pretraining and computational efficiency. The image and audio experiments primarily involve short continuation training from pretrained checkpoints rather than full self-supervised pretraining. Consequently, the benefits of K-JEPA in standard end-to-end JEPA pretraining remain insufficiently demonstrated. In addition, training-time and memory overhead should be reported to substantiate the claimed efficiency of the method.
6. Unclear relationship between target prediction and downstream performance. Some substantial improvements in target retrieval lead to only modest downstream gains. For example, Audio-JEPA's R@1 on SpeechCommands increases from 14.1% to 95.3%, while downstream accuracy improves by only 2.2 percentage points.
Questions for authors
Please see the Weaknesses section above.
8 comments
YTpy.a2 continuation: We additionally evaluated random-feature dimension sensitivity on across three SM-MNIST hosts in comparable settings:
Table B. This table shows the random-feature dimension sensitivity check on SM-MNIST. Safety averages Error-AUROC and (1 − AURC), while State error averages normalized MAE and RMSE. All results follow the training and evaluation described in the manuscript.
| Host | D | Macro-F1 ↑ | AUPRC ↑ | Safety ↑ | State error ↓ | R@10 ↑ | MRR ↑ | NLL ↓ |
|---|---|---|---|---|---|---|---|---|
| B-JEPA+K-JEPA | 64 | 43.2 | 25.3 | 62.5 | 0.44 | 7.3 | 4.1 | 0.56 |
| 256 | 43.5 | 27.6 | 65.6 | 0.44 | 10.7 | 6.2 | 0.56 | |
| 1024 | 43.0 | 27.7 | 66.4 | 0.44 | 8.3 | 4.3 | 0.56 | |
| V-JEPA+K-JEPA | 64 | 42.9 | 25.3 | 62.5 | 0.44 | 7.0 | 4.0 | 0.56 |
| 256 | 51.9 | 41.0 | 78.0 | 0.41 | 10.9 | 6.0 | 0.52 | |
| 1024 | 42.9 | 25.1 | 62.6 | 0.44 | 7.2 | 4.1 | 0.56 | |
| VJE+K-JEPA | 64 | 43.2 | 28.0 | 68.2 | 0.43 | 9.3 | 5.1 | 0.57 |
| 256 | 43.6 | 32.9 | 71.0 | 0.44 | 10.7 | 5.5 | 0.55 | |
| 1024 | 43.5 | 33.2 | 70.4 | 0.42 | 9.1 | 5.7 | 0.57 |
The standard configuration shows strongest overall performance and distribution quality, and performs particularly well for V-JEPA. B-JEPA exhibits smaller variations, while VJE shows metric-dependent sensitivity, with achieving better NLL, selective safety, and target prediction, and achieving slightly better AUPRC, state error, and MRR.
Finally, we investigated fixed versus adaptive loss weighting. Adaptive gradient balancing is motivated by the different gradient magnitudes of native JEPA and kernel-prediction losses (Section 3.2). To assess fixed weighting, we trained K-JEPA with a constant coefficient and compared checkpoints at epochs 10 and 20.
Table C. Fixed-weight training sensitivity on SM-MNIST. The results show within-run comparison of checkpoints at epochs 10 and 20 under a fixed kernel-loss coefficient . Safety averages Error-AUROC and (1 − AURC).
| Host | Epoch | Acc ↑ | Macro-F1 ↑ | AUPRC ↑ | NLL ↓ | Safety ↑ | State error ↓ |
|---|---|---|---|---|---|---|---|
| B-JEPA+ Fixed-weight K-JEPA | 10 | 75.56 | 56.23 | 43.94 | 0.4997 | 79.73 | 0.4023 |
| 20 | 75.34 | 53.34 | 42.34 | 0.5094 | 78.46 | 0.4077 | |
| C-JEPA+ Fixed-weight K-JEPA | 10 | 75.05 | 42.87 | 27.60 | 0.5610 | 66.22 | 0.4268 |
| 20 | 75.05 | 42.87 | 26.87 | 0.5616 | 65.04 | 0.4296 | |
| V-JEPA+ Fixed-weight K-JEPA | 10 | 74.96 | 52.32 | 41.07 | 0.5146 | 78.32 | 0.4060 |
| 20 | 74.90 | 44.74 | 33.81 | 0.5445 | 72.54 | 0.4239 |
The fixed-weight experiments show deterioration during continued training, with the magnitude depending on the host. The clearest pattern appears for V-JEPA, where Macro-F1 decreases from 52.32 to 44.74, AUPRC from 41.07 to 33.81, and selective safety from 78.32 to 72.54, alongside worsening NLL and state error. These results support the importance of adaptive loss balancing rather than using a fixed coefficient across training.
YTpy.q3. Modest and inconsistent downstream improvements [...] the improvements on ImageNet-100 are only 1.1 and 1.0 percentage points [...] LeJEPA slightly decreases from 88.0% to 87.9% [...] more comprehensive statistical evidence [is needed].
YTpy.a3: We clarify that K-JEPA's central contribution is distribution-aware latent prediction, not simply maximizing downstream classification accuracy. On ImageNet-100, all three image methods improve target R@1, MRR, and NLL, with further improvements in Brier and selective AURC for LpJEPA and LeJEPA, while I-JEPA remains at already low values (Table 4). Importantly, these gains also extend to external zero-shot retrieval on Copydays, where all nine model–missingness settings improve R@1, MRR, NLL, and ECE, with Brier improving or remaining unchanged (Table 5). These predictive and reliability benefits are achieved while preserving or improving downstream representation quality. All three image methods improve downstream Brier and ECE, accuracy improves for I-JEPA and LpJEPA, and LeJEPA's accuracy changes by only 0.1 points while improving AUPRC and Error-AUROC (Table 3). Thus, the image results demonstrate consistent benefits across target prediction, probabilistic scoring, calibration, and selective reliability, which cannot be captured by classification accuracy alone. This broader pattern also extends across audio, tabular, and world-model experiments (Tables 2, 6–8; Appendix Tables 11–12).
In addition, the improvements extend consistently across other modalities and tasks. K-JEPA improves accuracy, macro-F1, and AUPRC across all six audio model-dataset comparisons (Table 2), and downstream accuracy on all three tabular datasets (Table 6). Tables 11–13 also report various statistically significant gains across multiple tasks and JEPA formulations. Our results show consistent benefits across multiple performance and reliability metrics, even when individual accuracy gains are modest.
We report the paired ImageNet-100 results in the next comment.
YTpy.a3 continuation: To further address statistical uncertainty, we conducted paired ImageNet-100 evaluations on 5,000 matched examples using 1,000 bootstrap resamples. The following table (Table D) reports downstream differences between native JEPA and K-JEPA, along with estimated standard errors.
Table D. Differences (Δ) are K-JEPA minus native JEPA, calculated from the published manuscript Table 3 results. Standard errors (SEs) were estimated using 5,000 matched evaluation examples and 1,000 paired bootstrap resamples. Accuracy SEs were derived from paired correctness counts, while the remaining SEs were approximated from paired-bootstrap 95% confidence-interval widths. An asterisk (*) shows a favorable K-JEPA improvement over native JEPA exceeding one estimated standard error (SE).
| Metric | K-JEPA vs. I-JEPA | SE | K-JEPA vs. LpJEPA | SE | K-JEPA vs. LeJEPA | SE |
|---|---|---|---|---|---|---|
| Accuracy ↑ | +1.1* | 0.46 | +1.0* | 0.56 | -0.1 | 0.29 |
| Macro-F1 ↑ | +0.2 | 0.44 | +0.9* | 0.59 | -0.1 | 0.28 |
| AUPRC ↑ | -0.2 | 0.32 | +0.6* | 0.42 | +0.6* | 0.13 |
| Brier ↓ | -0.02* | 0.004 | -0.01* | 0.005 | -0.01* | 0.002 |
| NLL ↓ | +0.01 | 0.009 | -0.02* | 0.013 | 0.00 | 0.004 |
| ECE ↓ | -0.33 | 0.49 | -0.50 | 0.57 | -0.09 | 0.33 |
Across these results, K-JEPA improves 13 of 18 metrics, including all six for Rectified LpJEPA. Our additional paired evaluations corroborate this favorable pattern, with several improvements exceeding one estimated standard error. Together with the consistent target-prediction, calibration, and reliability improvements in Tables 4–5, these findings reinforce that K-JEPA's contribution extends beyond isolated downstream accuracy gains.
YTpy.q4. The distribution-identification guarantee applies to the population characteristic kernel, whereas the practical implementation uses finite random features [...] improvements in retrieval and downstream classification do not directly establish accurate recovery of multimodal conditional target distributions [...] more direct distributional evaluations and sensitivity analyses are needed.
YTpy.A4: We clarify that the finite-feature approximation is already analyzed in Proposition 4 (Appendix A.1.2), which bounds the absolute difference between the squared finite-feature and characteristic-kernel MMDs by .d. Furthermore, our evaluations are not limited to retrieval and classification accuracy. Tables 4, 5, 6, and 8 also evaluate Brier and NLL (probabilistic prediction quality), ECE (calibration), and AURC and selective risk (uncertainty reliability), alongside conformal coverage and prediction-set size in Table 8. Our results provide complementary evidence of predictive quality and reliability.
To directly assess conditional-distribution recovery, we additionally evaluated empirical conditional mean embedding (CME) discrepancy on SM-MNIST. For each of 256 contexts, we sampled 32 possible futures and estimated the empirical conditional kernel mean. The learned K-JEPA kernel predictor achieves substantially lower squared discrepancy than the transformed point-prediction reference: 30.52 versus 4.26 for B-JEPA (86.0% reduction) and 169.99 versus 7.77 for V-JEPA (95.4% reduction). The kernel predictor additionally achieves lower discrepancy across all 256 evaluated contexts for both hosts, which provides direct evidence of improved empirical conditional kernel-mean alignment.
We provide further ablations in our next comment.
We thank the reviewer for their helpful comments.
YTpy.q1. [...] theoretical results largely follow from established properties of conditional mean regression and characteristic kernel mean embeddings. The practical method mainly introduces an additional kernel-feature prediction loss into existing JEPA architectures. The authors should better clarify the novelty beyond applying established [...] techniques to JEPA.
YTpy.a1: We clarify that our contribution is not the introduction of conditional mean embeddings or characteristic kernels themselves, which are established tools [5]. K-JEPA's nontrivial methodological novelty lies in integrating distribution-identifying prediction directly into JEPA's shared representation learning, which makes a conditional kernel mean embedding the auxiliary predictor's population-optimal target (Corollary 1).
Unlike existing neural-CME methods [3], K-JEPA incorporates this principle directly into self-supervised representation learning and improves the learned representations while keeping the original JEPA objective and architecture (Eq. 11). It also differs from KerJEPA [4], which uses kernel discrepancies to regularize representations, and from variational, diffusion-based, or branching JEPA extensions (Table 1) which require additional assumptions or prediction mechanisms.
Our theoretical analysis further establishes how adaptive balancing and finite random features affect the distributional guarantees (Propositions 3-4). Beyond the method itself, we introduce a common evaluation of JEPA's native target prediction and reliability across various architectures, alongside downstream evaluation (Tables 2–8). Our contributions concern both a practical way to extend JEPA from point to distributional prediction and a systematic evaluation of its effects.
YTpy.q2. The paper lacks comprehensive ablations to isolate the contribution of the proposed distributional prediction mechanism. In particular, comparisons with parameter-matched auxiliary prediction heads, alternative nonlinear target transformations, and different kernel-feature dimensions are needed.
YTpy.a2: To isolate the contribution of the proposed distribution-sensitive prediction mechanism, we additionally evaluated various target-prediction controls under matched training and across five JEPA hosts:
Table A. Comparison of native JEPA, auxiliary MSE, random linear, random nonlinear, and K-JEPA objectives using downstream performance, calibration, and selective-prediction metrics. World-model results use SM-MNIST, while I-JEPA uses ImageNet-100. All results follow the corresponding evaluations described in the manuscript.
Across JEPA hosts, K-JEPA achieves the highest downstream Macro-F1 on four and the highest AUPRC on three, with improvements in probabilistic prediction and selective reliability. The strongest differences appear for V-JEPA, where K-JEPA improves Macro-F1 to 51.9, AUPRC to 41.0, Error AUROC to 70.0, and AURC to 14.0, clearly outperforming all auxiliary objectives. The relative advantages vary across hosts and metrics, but the complete evaluation shows that generic supervision does not consistently reproduce K-JEPA's strength.
We provide further ablations in our next comment to the reviewer.