PowerJEPA: From Tail-Aware Alignment to the Downstream Utility of Global Spectral Structure

Learning general-purpose embeddings requires preserving semantic correspondences while retaining variation useful across downstream tasks. Despite LeJEPA’s success in visual representation learning, its direct extension to general-purpose text embeddings yields limited downstream performance in our experiments. Empirical analysis identifies insufficient alignment of difficult positive pairs as a key limitation. Motivated by this finding, we introduce PowerJEPA, combining tail-aware power-mean alignment, selective hard-negative discrimination, and Sketched Isotropic Gaussian Regularization (SIGReg) to shape local and global embedding geometry. Experiments across text (MS MARCO and NLI) and image (ImageNet) modalities demonstrate its effectiveness. We further examine how the global representational capacity shaped by SIGReg, measured by effective dimension, relates to downstream performance. Spectral interventions on frozen text representations show that retrieval is more sensitive to truncation than clustering and reranking, while larger effective dimension does not consistently improve performance. Theory further shows that spectra with identical effective dimension can have different regularized effective degrees of freedom. We therefore combine effective dimension with spectral curvature into a capacity–shape description for assessing downstream utility. A calibrated predictor using both descriptors achieves a Spearman correlation of 0.97 with mean MTEB scores. These results connect geometric representation learning with calibrated diagnostics of downstream transfer.