acceptodds
Under review as a conference paper at ICLR 2027

PowerJEPA: From Tail-Aware Alignment to the Downstream Utility of Global Spectral Structure

Abstract

Learning general-purpose embeddings requires preserving semantic correspondences while retaining variation useful across downstream tasks. Despite LeJEPA’s success in visual representation learning, its direct extension to general-purpose text embeddings yields limited downstream performance in our experiments. Empirical analysis identifies insufficient alignment of difficult positive pairs as a key limitation. Motivated by this finding, we introduce PowerJEPA, combining tail-aware power-mean alignment, selective hard-negative discrimination, and Sketched Isotropic Gaussian Regularization (SIGReg) to shape local and global embedding geometry. Experiments across text (MS MARCO and NLI) and image (ImageNet) modalities demonstrate its effectiveness. We further examine how the global representational capacity shaped by SIGReg, measured by effective dimension, relates to downstream performance. Spectral interventions on frozen text representations show that retrieval is more sensitive to truncation than clustering and reranking, while larger effective dimension does not consistently improve performance. Theory further shows that spectra with identical effective dimension can have different regularized effective degrees of freedom. We therefore combine effective dimension with spectral curvature into a capacity–shape description for assessing downstream utility. A calibrated predictor using both descriptors achieves a Spearman correlation of 0.97 with mean MTEB scores. These results connect geometric representation learning with calibrated diagnostics of downstream transfer.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation learning, JEPA, Tail-aware alignment, Spectral geometry

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2GeSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

PowerJEPA combines tail-aware power-mean alignment, selective hard-negative discrimination, and SIGReg to learn representations, with experiments on MS MARCO, AllNLI, and ImageNet-1K. The paper further investigates task-dependent spectral utility through frozen-encoder spectral interventions and develops a capacity–shape description combining effective dimension (ED) and spectral curvature, which improves calibrated prediction of downstream task means on 12 held-out configurations.

Critical strengths
  1. The power-mean objective explicitly increases the relative distance-gradient weights assigned to difficult positive pairs. The paper examines this mechanism through positive-distance tails, sampled ranking, and component analysis, rather than reporting only aggregate downstream performance.

  2. The analysis of task-dependent spectral utility provides useful insight beyond effective dimension alone. The frozen-encoder truncation study shows that retrieval is more sensitive to spectral truncation than clustering and reranking, and adding spectral curvature to ED improves Spearman correlation from 0.692 to 0.972 and reduces RMSE from 2.034 to 0.753 on 12 held-out configurations, providing evidence of complementary predictive information within the evaluated settings.

Critical weaknesses
  1. The independent contribution of power-mean alignment is not established against strong baselines.

    • On MS MARCO (Table 1), PowerJEPA without hard negatives (54.31) is 1.65 points below InfoNCE (55.96). With hard negatives, PowerJEPA (55.98) remains below InfoNCE + hard negatives (56.72) and is only 0.12 points above TERM + SIGReg + hard negatives (55.86).
    • The comparison between calibrated PowerJEPA (56.81) and uncalibrated InfoNCE (56.72) does not isolate the training objective from post-hoc spectral calibration. Appendix B.5 states that the calibration study uses its own inference implementation and 37-task aggregation, that "its absolute means should not be compared directly with ... the main table's 39-task aggregation," and that "these results do not establish superiority over calibrated InfoNCE."
    • On AllNLI (Table 2), the 0.25-point average advantage over InfoNCE (75.49 vs. 75.24) is driven largely by a single task, STS22.v2 (+9.73). The image results are similarly small (e.g., +0.13 CMC@5 in Table 4), and no variance is reported.
  2. The incremental contribution of the capacity–shape description requires clearer positioning against prior work.
    $\alpha $-ReQ [1] already investigates representation quality through eigenspectrum decay and explicitly acknowledges that a favorable decay coefficient alone is not sufficient for good downstream performance. Although the paper includes an ED + single slope baseline (Table 5), it neither cites $\alpha $-ReQ nor specifies how this baseline relates to it, leaving the incremental contribution of combining effective dimension with spectral curvature insufficiently contextualized.

[1] Agrawal K K, Mondal A K, Ghosh A, et al. $\alpha $-ReQ: Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay. Advances in Neural Information Processing Systems, 2022, 35: 17626-17638.

Questions for authors

Clarification requests and targeted validation

  1. Independent contribution of power-mean alignment: Which training configurations and hard-negative supervision settings are shared across PowerJEPA, InfoNCE, and TERM, and how are their tuning budgets matched?

  2. Equivalent post-hoc spectral calibration: How does InfoNCE perform when given the same post-hoc spectral calibration procedure, access to calibration data, and development-based selection budget as PowerJEPA, evaluated on the same task set with the same inference implementation?

  3. Relationship to $\alpha $-ReQ: Does ED + single slope use the decay-coefficient estimation procedure of $\alpha $-ReQ [1]? Please clarify their methodological relationship and explain what additional information the capacity–shape description captures beyond this baseline.

Minor correction

On line 414, "PowerJEPA consistently outperforms Lepanto" should read "PowerJEPA consistently outperforms LeJEPA".

3 comments

Thank you very much for your time and feedback, which have helped us greatly improve the quality of our paper.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version's numbering.

[W1.1 Independent contribution and performance against strong baselines]

Our main contribution is to adapt the JEPA framework to text representation learning and address the limitations of its original formulation in this setting. The relevant LeJEPA comparisons show improvements of 4.61 points without explicit hard negatives and 2.48 points with hard negatives on MS MARCO. PowerJEPA also improves over TERM + SIGReg by 1.37 points on AllNLI.

[W1.2 & Q2 Post-hoc calibration] The table below reports results with the same post-hoc calibration applied to InfoNCE and TERM + SIGReg. Our main contribution is to adapt the JEPA framework to text representation learning and address the limitations of its original formulation in this setting. We do not claim uniform superiority over strong text-embedding baselines such as contrastive learning. We have updated the corresponding discussion in the main text to report these matched-calibration results (page 9 of the revised paper).

MethodUncalibratedMatched calibrationGain
InfoNCE56.7657.50+0.74
TERM + SIGReg55.5856.55+0.97
PowerJEPA55.9756.90+0.93

[W1.3 Multiple-seed evaluations]

We supplemented the original seed-42 experiments with seeds 43 and 44. The tables below report the mean ± standard deviation across all three seeds. We have updated Tables 1 and 2 in the main text to report results averaged over three seeds (P. 7 of the revised paper).

MS MARCO

MethodCls.Clust.PairRerankRetr.STSSum.Avg.
InfoNCE63.89 ± 0.1644.69 ± 0.1873.87 ± 1.0943.60 ± 0.5341.96 ± 0.2572.90 ± 1.3222.63 ± 0.4855.64 ± 0.45
InfoNCE + hard negatives65.33 ± 0.2643.96 ± 0.0876.06 ± 0.5843.86 ± 0.2143.70 ± 0.3873.45 ± 0.2024.60 ± 0.1956.51 ± 0.13
LeJEPA (w/ projector)53.97 ± 0.0947.06 ± 0.3152.21 ± 0.7040.60 ± 0.1827.94 ± 1.2765.00 ± 0.0314.35 ± 1.7447.25 ± 0.38
LeJEPA (w/o projector)57.92 ± 5.4946.39 ± 1.2760.88 ± 11.5642.00 ± 2.1633.51 ± 6.6267.60 ± 3.6417.18 ± 5.7350.52 ± 4.24
LeJEPA (w/o projector) + hard negatives61.74 ± 0.0845.48 ± 0.0169.08 ± 0.0343.54 ± 0.0238.20 ± 0.0170.18 ± 0.0220.49 ± 1.0453.49 ± 0.03
VISReg + hard negatives63.00 ± 1.6843.76 ± 0.3670.73 ± 0.0642.29 ± 1.3434.88 ± 4.6669.59 ± 2.3525.20 ± 2.4552.65 ± 1.39
TERM + SIGReg + hard negatives64.10 ± 0.2645.18 ± 0.5973.12 ± 0.9643.45 ± 0.0741.72 ± 0.6672.17 ± 0.4726.30 ± 1.0055.58 ± 0.39
PowerJEPA (ours)63.54 ± 2.3045.64 ± 1.0370.61 ± 3.9743.13 ± 0.7141.58 ± 1.5671.51 ± 1.2424.41 ± 1.0755.13 ± 1.16
PowerJEPA (ours) + hard negatives65.04 ± 0.1745.09 ± 0.2573.63 ± 0.2943.65 ± 0.0442.46 ± 0.3172.49 ± 0.1524.02 ± 0.5155.97 ± 0.02

AllNLI

MethodSTS12STS13STS14STS15STS17STS22.v2STS-BSICK-RBIOSSESAvg.
InfoNCE70.18 ± 0.3880.15 ± 0.2474.04 ± 0.5683.05 ± 0.6785.29 ± 0.8747.37 ± 0.0682.07 ± 0.5176.94 ± 0.3475.42 ± 0.0774.95 ± 0.41
LeJEPA (w/ projector)59.70 ± 0.2365.80 ± 0.0959.63 ± 0.1873.84 ± 0.2877.68 ± 0.2442.62 ± 1.2468.11 ± 0.1558.20 ± 0.0772.05 ± 0.1164.18 ± 0.06
LeJEPA (w/o projector)‡68.31 ± 0.1180.02 ± 0.0972.72 ± 0.0381.46 ± 0.0383.21 ± 0.3456.78 ± 0.7478.17 ± 0.1573.95 ± 0.0577.51 ± 0.3274.68 ± 0.16
TERM + SIGReg‡70.55 ± 0.1080.11 ± 0.2572.74 ± 0.0981.62 ± 0.0081.51 ± 0.1652.70 ± 1.3877.62 ± 0.3773.48 ± 0.4277.34 ± 0.1074.19 ± 0.23
PowerJEPA (ours)72.50 ± 0.2782.60 ± 0.1174.95 ± 0.1782.40 ± 0.0282.70 ± 0.1157.15 ± 0.0078.56 ± 0.0672.91 ± 0.2776.22 ± 1.0675.56 ± 0.09

[W2. Positioning against α\alpha-ReQ]

Thank you for pointing out this relevant work. We cited α\alpha-ReQ and clarify that spectral decay as a representation-quality diagnostic is an established idea. Our incremental contribution is to examine whether head-to-tail spectral curvature provides additional predictive information beyond effective dimension and a single decay slope, supported by the comparisons in Table 5. We have revised Section 2 to cite α\alpha-ReQ and Section 6.4 to clarify our focus on the additional predictive value of curvature beyond effective dimension and a single decay slope.

[Q1. Shared training configurations, hard negatives, and tuning budgets]

All methods share the same network architecture and common experimental configuration. Learning rate and batch size are tuned independently for each method, and we report the best-performing configuration found in each search. In the explicit-hard-negative comparisons, all methods receive hard-negative supervision; InfoNCE additionally uses in-batch negatives. For example, the contrastive learning results across different learning rates and batch sizes are shown below.

Batch settingLRHard negativesCls.Clust.PairRerankRetr.STSSum.Avg.
81922 × 10⁻⁵165.2143.5274.5742.3837.4871.1326.3554.28
10242 × 10⁻⁵165.5244.0175.6543.7043.4373.3024.4756.41
2562 × 10⁻⁵165.3442.7973.6943.3740.9471.9424.9855.09
1282 × 10⁻⁵165.2842.1273.1643.0439.8071.3725.8054.51
2561 × 10⁻⁵No63.7844.8173.1043.2241.7871.9622.9755.32
2562 × 10⁻⁵No65.2244.0173.2841.6534.1170.1427.6653.27
81922 × 10⁻⁵No64.1244.0674.1342.7339.3371.5026.3954.69

[Q3. Relationship between the single-slope baseline and α\alpha-ReQ]

A single slope summarizes spectral decay over a fitted interval. Our curvature statistic captures the difference between decay slopes over ranks 8–128 and 129–512, describing head-to-tail variation that a single fitted slope does not explicitly represent. Under the same prediction protocol, replacing the single-slope descriptor with curvature alongside ED improves Spearman correlation from 0.902 to 0.972 and reduces RMSE from 1.570 to 0.753. This supports the additional predictive value of curvature within the evaluated configurations.

[Minor correction]

Thank you for identifying the typo. We replaced “Lepanto” with “LeJEPA”.

Thank you for the additional experiments. The matched calibration and three-seed results make the paper's positioning clearer, but they also show that PowerJEPA still trails InfoNCE. I find the contribution relatively limited, and I maintain my rating.

Reviewer_LvZKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PowerJEPA, an extension of LeJEPA for representation learning, particularly text embeddings. Motivated by the observation that MSE-based alignment leaves difficult positive pairs insufficiently aligned, the authors introduce a power-mean alignment objective that assigns greater relative importance to large positive-pair distances. The proposed framework combines this objective with additional selective hard-negative discrimination and SIGReg. Beyond the learning objective, the paper investigates the relationship between global embedding spectra and downstream performance. Through PCA truncation and spectral reweighting, the authors demonstrate that different downstream tasks exhibit different sensitivities to representation dimensionality.

Critical strengths
  1. The paper identifies a meaningful limitation of MSE-based alignment in text representation learning: a subset of difficult positive pairs can remain poorly aligned despite optimization of the average alignment objective.

  2. The empirical results provide interesting evidence that improving positive alignment does not necessarily improve downstream performance. This observation illustrates an important tension between local alignment and global representation diversity.

  3. The investigation of spectral curvature beyond effective dimension is interesting. The comparison with alignment, uniformity, and single-slope spectral descriptors provides useful empirical evidence that spectral shape may contain complementary information about downstream performance.

Critical weaknesses
  1. The main methodological novelty is replacing MSE alignment with a power-mean objective. However, its underlying mechanism is a form of hard-example reweighting, which is closely related to established robust optimization and sample weighting techniques. Equation (6) follows directly from differentiating the power mean, while Proposition 4.1 is derived from standard Lp duality, Markov's inequality, and Hölder's inequality. These results provide useful intuition but do not establish a fundamentally new optimization principle.

  2. The proposed objective does not consistently outperform standard contrastive learning. In Table 1, PowerJEPA achieves an average MTEB score of 54.31 which is below the InfoNCE baseline of 55.96. PowerJEPA + hard negatives achieves 55.98, below InfoNCE with hard negatives at 56.72.Moreover, the post-hoc calibrated PowerJEPA result of 56.81 is only marginally better than the uncalibrated InfoNCE baseline, and the same calibration procedure is not applied to InfoNCE. On AllNLI experiment, the improvement over InfoNCE is also insignificant (75.49 versus 75.24 in Table 2). Although PowerJEPA significantly improves upon LeJEPA, this primarily establishes that power-mean alignment is more effective than the MSE-based adaptation in the investigated setting. It does not establish a clear practical advantage over established contrastive-learning objectives.

  3. The paper presents the core training objective as a combination of power-mean alignment, selective hard-negative discrimination, and SIGReg. However, Appendix B.1 reveals that the main MS MARCO training recipe additionally incorporates an L2 triplet loss, self-hard-negative symmetric supervision, a global hard-document softplus term, additional query regularization, and a residual output MLP with a scheduled coefficient. These components may substantially affect both positive-pair alignment and negative-relative ranking, making it difficult to attribute the observed improvements. Although Table 3 includes component ablations, it does not fully disentangle the effects of these auxiliary mechanisms.

  4. Proposition 5.1 derives a bias–variance decomposition for a diagonal shrinkage estimator and establishes that retaining a spectral direction is beneficial when its task-specific signal exceeds a noise-dependent threshold. However, this analysis also reveals a fundamental limitation: downstream utility depends not only on the covariance spectrum but also on the allocation of task-specific signals across spectral directions. The proposed effective dimension and spectral curvature descriptors depend only on the embedding covariance spectrum and do not explicitly capture these task-specific signals. Consequently, the theoretical results do not explain why these particular spectral descriptors should reliably predict downstream utility, nor do they establish a direct connection to the retrieval, clustering, and semantic similarity tasks evaluated in the paper. The theoretical analysis is therefore better understood as motivation for investigating spectral properties rather than a theoretical justification for the proposed predictor.

Questions for authors

Please see weaknesses.

1 comment

We sincerely appreciate your time and thoughtful feedback. Your suggestions have greatly improved the quality of our paper.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version's numbering.

[W1. Methodological novelty]

To the best of our knowledge, this is the first work to systematically identify and empirically demonstrate difficult-positive misalignment in non-contrastive text representation learning. Our key contribution is therefore to identify and address difficult-positive misalignment in the LeJEPA text adaptation and examine its interaction with representation diversity and downstream transfer, rather than to propose a new optimization method. Equation (6) and Proposition 4.1 explain the objective’s gradient allocation and tail control using standard mathematical tools. TERM provides a direct comparison with an established reweighting approach.

[W2. Performance against strong baselines]

Our main contribution is to adapt JEPA to text representation learning and address limitations of its original formulation. The matched LeJEPA comparisons show improvements of 4.61 and 2.48 points without and with explicit hard negatives on MS MARCO. PowerJEPA also improves over TERM + SIGReg by 4.63 points on AllNLI. The small difference from InfoNCE on AllNLI is presented as competitive performance.

[W3. Attribution of performance gains]

The auxiliary losses and architectural components in Appendix B.1 are shared across the matched LeJEPA, TERM, and PowerJEPA configurations. We have revised Appendix B.1 to make this explicit. Only the alignment objective differs: MSE, TERM, or power mean. Our ablations likewise change only the specified component while keeping other settings fixed. Therefore, the matched comparisons isolate the contribution of the alignment objective.

[W4. Spectral curvature and downstream utility]

Proposition 5.1 establishes task-signal dependence but does not directly justify curvature. To address this gap, we add a conditional theorem after Proposition 5.1: under a two-slope spectral model, isotropic task-signal second moments, and fixed dimension and relative regularization, ED and curvature jointly determine average regularized prediction risk; this risk generally changes with curvature even at fixed ED.

This establishes a conditional connection between curvature and prediction risk. Prediction of downstream benchmark scores remains an empirical claim, supported by the 12 configurations excluded from descriptor selection and predictor fitting. The newly added theorem is stated below.

Spectral curvature and prediction risk. We next establish a conditional link between head-to-tail spectral curvature and the prediction risk in Proposition 5.1. Fix d≥4d \geq 4, a breakpoint 2≤m≤d−22 \leq m \leq d-2, and increasing spectral-regression abscissae x1<⋯<xdx_1 < \cdots < x_d. Consider normalized covariance spectra of the form

pj(α,κ)=exp⁡[−αxj−κ(xj−xm)+]∑ℓ=1dexp⁡[−αxℓ−κ(xℓ−xm)+],\displaystyle p_j(\alpha,\kappa)=\frac{\exp[-\alpha x_j-\kappa(x_j-x_m)_+]}{\sum_{\ell=1}^{d}\exp[-\alpha x_\ell-\kappa(x_\ell-x_m)_+]},

where α>0\alpha > 0, α+κ>0\alpha+\kappa > 0, and pj=λj/tr⁡(C)p_j=\lambda_j/\operatorname{tr}(C). The head and tail slopes of the log-spectrum are −α-\alpha and −(α+κ)-(\alpha+\kappa), respectively, so their difference equals κ\kappa. The abscissae are fixed by the spectral-fitting convention; for log–log fits, xj=log⁡jx_j=\log j.

Under the observation model in Proposition 5.1, additionally average over a task ensemble independent of the noise, with Etask[aj2]=s2\mathbb{E}_{\mathrm{task}}[a_j^2]=s^2 for every jj. Assume s2,ν≥0s^2,\nu \geq 0 and s2+ν>0s^2+\nu > 0. At a fixed relative regularization scale ρ=η/tr⁡(C)>0\rho=\eta/\operatorname{tr}(C) > 0, the average risk per coordinate is

Rρ(p)=1dE∥a^−a∥22=1d∑j=1ds2ρ2+νpj2(pj+ρ)2.\displaystyle \mathcal{R}_\rho(p)=\frac{1}{d}\mathbb{E}\lVert\widehat{a}-a\rVert_2^2=\frac{1}{d}\sum_{j=1}^{d}\frac{s^2\rho^2+\nu p_j^2}{(p_j+\rho)^2}.

Theorem 5.2 (Curvature contains risk information beyond effective dimension). Fix d,m,(xj),s2,νd,m,(x_j),s^2,\nu as above. Within this spectral family:

(i) Every attainable pair (Deff,κ)(D_{\mathrm{eff}},\kappa) uniquely determines the normalized spectrum. Consequently, for each ρ>0\rho > 0, there exists a function FρF_\rho such that

Rρ(p)=Fρ!(log⁡Deff(p),κ).\displaystyle \mathcal{R}_\rho(p)=F_\rho\\!\left(\log D_{\mathrm{eff}}(p),\kappa\right).

(ii) At every admissible parameter point (α0,κ0)(\alpha_0,\kappa_0), there exists a local smooth curve α=α(κ)\alpha=\alpha(\kappa) along which DeffD_{\mathrm{eff}} remains constant. At that point,

ddκRρ!(p(α(κ),κ))∣κ=κ0≠0\displaystyle \left.\frac{\mathrm{d}}{\mathrm{d}\kappa}\mathcal{R}_\rho\\!\left(p(\alpha(\kappa),\kappa)\right)\right|_{\kappa=\kappa_0}\neq 0

for all but finitely many ρ>0\rho > 0.

Reviewer_xA4FOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

They first identify a problem where LeJepa performs badly when used to create general-purpose text embeddings, unlike its success in visual representation learning. The main reason identified is insufficient alignment of difficult positive pairs. To address this issue, they introduce PowerJEPA using tail-aware power-mean alignment, selecting hard-negative discrimination and SIGReg for regularisation, the same as LeJepa. They show dependence of spectral changes in various tasks and modalities. It argues that effective dimension alone is insufficient to characterise representation quality and proposes spectral curvature as a complementary descriptor.

Critical strengths

The alignment objective is well motivated according to my understanding. The power-mean objective provides a simple, differentiable mechanism for emphasising larger residuals without explicit hard-positive mining. Significant improvement over LeJepa for text; proposed modifications are interesting and useful in this case. The observation that retrieval is more sensitive to truncation compared to other tasks is interesting and can be researched further. Ablations are useful and provide a sufficient justification for their method (Table 3)

Critical weaknesses

Improvements over oter baselines are ber less (InfoNCE and TERM); also,the post-hoc calibrated result is not directly comparable to the InfoNCE baseline, since the latter is not subjected to equivalent calibration. The actual training pipeline is considerably more complex than the proposed objective, Appendix B.1 (The p = 18 recipe combines normalized power alignment, union SIGReg with an additional query regularizer, L2 triplet loss with weight 5 and margin 0.3, self-hardnegative symmetric supervision with weight 0.5, and a global hard-document softplus term with weight 0.1, k = 8, cosine margin 0.05, and temperature 0.05. Its residual output MLP coefficient decays from 1.0 to 0.2. ) which is much more complex than defined. The theoretical results provide limited explanation of the empirical findings like - proposition 5.1 does not directly establish why the proposed spectral curvature statistic should predict downstream performance. Evaluation selection and statistical reliability clarification, mostl results seem single seed, hyperparam selection is unclear.

Questions for authors

Provide matched compute budget exps, multiple-seed evaluations, and equivalent post-hoc calibration for InfoNCE and TERM. How much of the reported improvement is caused by power-mean alignment, rather than the additional loss terms and architectural choices?

How well does the ED + curvature predictor generalize to unrelated pretrained embedding models, rather than variations of Qwen3-0.6B? Maybe Llama or gemma How were hyperparameters selected, any MTEB test scores used in selecting the final configuration? Variability of seeds must be provided, atleast for the major claims

4 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback, which has helped us to substantially improve the clarity of our paper.

We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version's numbering.

[Q1.1 Clarification on whether the performance gains come from power-mean alignment]

Thank you for raising this important question. We would like to clarify that the auxiliary losses and architectural components listed in Appendix B.1 are shared across the matched LeJEPA, TERM, and PowerJEPA configurations. We have revised Appendix B.1 to make this explicit. The only change is the alignment objective: MSE-based alignment for the LeJEPA adaptation, TERM-based alignment for the TERM baseline, and power-mean alignment for PowerJEPA. Similarly, our ablation experiments change only the specified component while keeping all other settings fixed. Therefore, the reported improvements in the matched alignment comparisons can be attributed to power-mean alignment. The relevant LeJEPA comparisons show improvements of 4.61 points without explicit hard negatives and 2.48 points with hard negatives on MS MARCO. PowerJEPA also improves over TERM + SIGReg by 1.37 points on AllNLI.

[Q1.2 Multiple-seed evaluations]

We supplemented the original seed-42 experiments with seeds 43 and 44. The tables below report the mean ± standard deviation across all three seeds. We have updated Tables 1 and 2 in the main text to report results averaged over three seeds (P. 7 of the revised paper).

MS MARCO

MethodCls.Clust.PairRerankRetr.STSSum.Avg.
InfoNCE63.89 ± 0.1644.69 ± 0.1873.87 ± 1.0943.60 ± 0.5341.96 ± 0.2572.90 ± 1.3222.63 ± 0.4855.64 ± 0.45
InfoNCE + hard negatives65.33 ± 0.2643.96 ± 0.0876.06 ± 0.5843.86 ± 0.2143.70 ± 0.3873.45 ± 0.2024.60 ± 0.1956.51 ± 0.13
LeJEPA (w/ projector)53.97 ± 0.0947.06 ± 0.3152.21 ± 0.7040.60 ± 0.1827.94 ± 1.2765.00 ± 0.0314.35 ± 1.7447.25 ± 0.38
LeJEPA (w/o projector)57.92 ± 5.4946.39 ± 1.2760.88 ± 11.5642.00 ± 2.1633.51 ± 6.6267.60 ± 3.6417.18 ± 5.7350.52 ± 4.24
LeJEPA (w/o projector) + hard negatives61.74 ± 0.0845.48 ± 0.0169.08 ± 0.0343.54 ± 0.0238.20 ± 0.0170.18 ± 0.0220.49 ± 1.0453.49 ± 0.03
VISReg + hard negatives63.00 ± 1.6843.76 ± 0.3670.73 ± 0.0642.29 ± 1.3434.88 ± 4.6669.59 ± 2.3525.20 ± 2.4552.65 ± 1.39
TERM + SIGReg + hard negatives64.10 ± 0.2645.18 ± 0.5973.12 ± 0.9643.45 ± 0.0741.72 ± 0.6672.17 ± 0.4726.30 ± 1.0055.58 ± 0.39
PowerJEPA (ours)63.54 ± 2.3045.64 ± 1.0370.61 ± 3.9743.13 ± 0.7141.58 ± 1.5671.51 ± 1.2424.41 ± 1.0755.13 ± 1.16
PowerJEPA (ours) + hard negatives65.04 ± 0.1745.09 ± 0.2573.63 ± 0.2943.65 ± 0.0442.46 ± 0.3172.49 ± 0.1524.02 ± 0.5155.97 ± 0.02

AllNLI

MethodSTS12STS13STS14STS15STS17STS22.v2STS-BSICK-RBIOSSESAvg.
InfoNCE70.18 ± 0.3880.15 ± 0.2474.04 ± 0.5683.05 ± 0.6785.29 ± 0.8747.37 ± 0.0682.07 ± 0.5176.94 ± 0.3475.42 ± 0.0774.95 ± 0.41
LeJEPA (w/ projector)59.70 ± 0.2365.80 ± 0.0959.63 ± 0.1873.84 ± 0.2877.68 ± 0.2442.62 ± 1.2468.11 ± 0.1558.20 ± 0.0772.05 ± 0.1164.18 ± 0.06
LeJEPA (w/o projector)‡68.31 ± 0.1180.02 ± 0.0972.72 ± 0.0381.46 ± 0.0383.21 ± 0.3456.78 ± 0.7478.17 ± 0.1573.95 ± 0.0577.51 ± 0.3274.68 ± 0.16
TERM + SIGReg‡70.55 ± 0.1080.11 ± 0.2572.74 ± 0.0981.62 ± 0.0081.51 ± 0.1652.70 ± 1.3877.62 ± 0.3773.48 ± 0.4277.34 ± 0.1074.19 ± 0.23
PowerJEPA (ours)72.50 ± 0.2782.60 ± 0.1174.95 ± 0.1782.40 ± 0.0282.70 ± 0.1157.15 ± 0.0078.56 ± 0.0672.91 ± 0.2776.22 ± 1.0675.56 ± 0.09

[Q1.3 Post-hoc calibration]

Thank you for the suggestion. The table below reports results with the same post-hoc calibration applied to InfoNCE and TERM + SIGReg. Our main contribution is to adapt the JEPA framework to text representation learning and address the limitations of its original formulation in this setting. We do not claim uniform superiority over strong text-embedding baselines such as contrastive learning. We have updated the corresponding discussion in the main text to report these matched-calibration results (page 9, lines 455–474 of the revised paper).

MethodUncalibratedMatched calibrationGain
InfoNCE56.7657.50+0.74
TERM + SIGReg55.5856.55+0.97
PowerJEPA55.9756.90+0.93

[Q2 ED + curvature predictor generalize to unrelated pretrained embedding models]

Thank you for your suggestion. We report additional results on Llama and Gemma below, obtained using three random seeds (42, 43, and 44). These post-hoc cross-validation results support the complementary value of curvature observed on Qwen3-0.6B. We have added these additional cross-backbone experiments and the updated predictor evaluation to Appendix B.6.3 (page 22) of the revised manuscript.

PredictorLlama RMSE ↓Gemma RMSE ↓
ED1.3898.508
Curvature1.3981.389
Alignment + uniformity1.8611.282
ED + alignment1.3891.389
ED + uniformity2.6083.345
ED + single slope1.3897.034
ED + curvature1.0370.593

After adding Llama and Gemma, the updated correlation coefficients and RMSE values are shown in the table below.

PredictorSpearman ↑RMSE ↓
ED0.0673.938
Curvature0.7711.238
Alignment + uniformity0.8291.463
ED + alignment0.8291.354
ED + uniformity0.8481.769
ED + single slope0.3333.342
ED + curvature0.8861.030

[Hyperparameters selection]

Thank you for pointing this out. We selected the hyperparameters empirically through tuning, and Figure 5 reports the effects of varying the power exponent, batch size, margin, and loss weights. For all baselines, we use the same network architecture and other settings as PowerJEPA, while separately tuning hyperparameters such as the learning rate and batch size for each method. For example, the contrastive learning results across different learning rates and batch sizes are shown below.

Batch settingLRHard negativesCls.Clust.PairRerankRetr.STSSum.Avg.
81922 × 10⁻⁵165.2143.5274.5742.3837.4871.1326.3554.28
10242 × 10⁻⁵165.5244.0175.6543.7043.4373.3024.4756.41
2562 × 10⁻⁵165.3442.7973.6943.3740.9471.9424.9855.09
1282 × 10⁻⁵165.2842.1273.1643.0439.8071.3725.8054.51
2561 × 10⁻⁵No63.7844.8173.1043.2241.7871.9622.9755.32
2562 × 10⁻⁵No65.2244.0173.2841.6534.1170.1427.6653.27
81922 × 10⁻⁵No64.1244.0674.1342.7339.3371.5026.3954.69

[Theoretical justification]

Thank you for pointing out this gap. We have added Theorem 5.2 to Section 5, following Proposition 5.1, with the full proof and approximation bounds in Appendix A.5. Under a two-slope spectral model and a task ensemble with isotropic second moments, the theorem establishes that ED and curvature jointly determine average regularized prediction risk, and that varying curvature generally changes this risk even when ED is held fixed. This provides a direct, conditional explanation for the complementary information captured by curvature. The approximation bounds quantify deviations from these assumptions, while the relationship to downstream benchmark scores remains an empirical finding evaluated in Section 6.4. The newly added theorem is stated below.

Spectral curvature and prediction risk. We next establish a conditional link between head-to-tail spectral curvature and the prediction risk in Proposition 5.1. Fix d≥4d \geq 4, a breakpoint 2≤m≤d−22 \leq m \leq d-2, and increasing spectral-regression abscissae x1<⋯<xdx_1 < \cdots < x_d. Consider normalized covariance spectra of the form

pj(α,κ)=exp⁡[−αxj−κ(xj−xm)+]∑ℓ=1dexp⁡[−αxℓ−κ(xℓ−xm)+],\displaystyle p_j(\alpha,\kappa)=\frac{\exp[-\alpha x_j-\kappa(x_j-x_m)_+]}{\sum_{\ell=1}^{d}\exp[-\alpha x_\ell-\kappa(x_\ell-x_m)_+]},

where α>0\alpha > 0, α+κ>0\alpha+\kappa > 0, and pj=λj/tr⁡(C)p_j=\lambda_j/\operatorname{tr}(C). The head and tail slopes of the log-spectrum are −α-\alpha and −(α+κ)-(\alpha+\kappa), respectively, so their difference equals κ\kappa. The abscissae are fixed by the spectral-fitting convention; for log–log fits, xj=log⁡jx_j=\log j.

Under the observation model in Proposition 5.1, additionally average over a task ensemble independent of the noise, with Etask[aj2]=s2\mathbb{E}_{\mathrm{task}}[a_j^2]=s^2 for every jj. Assume s2,ν≥0s^2,\nu \geq 0 and s2+ν>0s^2+\nu > 0. At a fixed relative regularization scale ρ=η/tr⁡(C)>0\rho=\eta/\operatorname{tr}(C) > 0, the average risk per coordinate is

Rρ(p)=1dE∥a^−a∥22=1d∑j=1ds2ρ2+νpj2(pj+ρ)2.\displaystyle \mathcal{R}_\rho(p)=\frac{1}{d}\mathbb{E}\lVert\widehat{a}-a\rVert_2^2=\frac{1}{d}\sum_{j=1}^{d}\frac{s^2\rho^2+\nu p_j^2}{(p_j+\rho)^2}.

Theorem 5.2 (Curvature contains risk information beyond effective dimension). Fix d,m,(xj),s2,νd,m,(x_j),s^2,\nu as above. Within this spectral family:

(i) Every attainable pair (Deff,κ)(D_{\mathrm{eff}},\kappa) uniquely determines the normalized spectrum. Consequently, for each ρ>0\rho > 0, there exists a function FρF_\rho such that

Rρ(p)=Fρ!(log⁡Deff(p),κ).\displaystyle \mathcal{R}_\rho(p)=F_\rho\\!\left(\log D_{\mathrm{eff}}(p),\kappa\right).

(ii) At every admissible parameter point (α0,κ0)(\alpha_0,\kappa_0), there exists a local smooth curve α=α(κ)\alpha=\alpha(\kappa) along which DeffD_{\mathrm{eff}} remains constant. At that point,

ddκRρ!(p(α(κ),κ))∣κ=κ0≠0\displaystyle \left.\frac{\mathrm{d}}{\mathrm{d}\kappa}\mathcal{R}_\rho\\!\left(p(\alpha(\kappa),\kappa)\right)\right|_{\kappa=\kappa_0}\neq 0

for all but finitely many ρ>0\rho > 0.

Reviewer_xA4FOpenReview ↗

Response to Rebuttal

Thank you for the detailed response. The multi-seed results, the clarification that auxiliary components are shared across the matched configurations, the matched calibration for the baselines, and the Llama and Gemma predictor experiments do address several of my concerns. However, with three seeds and matched calibration, PowerJEPA remains below InfoNCE on MS MARCO on both, hard negative and the calibrated results and also, the gain over TERM + SIGReg is small. I therefore maintain my score as I believe the work needs more time before it is published.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.