- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PowerJEPA combines tail-aware power-mean alignment, selective hard-negative discrimination, and SIGReg to learn representations, with experiments on MS MARCO, AllNLI, and ImageNet-1K. The paper further investigates task-dependent spectral utility through frozen-encoder spectral interventions and develops a capacity–shape description combining effective dimension (ED) and spectral curvature, which improves calibrated prediction of downstream task means on 12 held-out configurations.
Critical strengths
-
The power-mean objective explicitly increases the relative distance-gradient weights assigned to difficult positive pairs. The paper examines this mechanism through positive-distance tails, sampled ranking, and component analysis, rather than reporting only aggregate downstream performance.
-
The analysis of task-dependent spectral utility provides useful insight beyond effective dimension alone. The frozen-encoder truncation study shows that retrieval is more sensitive to spectral truncation than clustering and reranking, and adding spectral curvature to ED improves Spearman correlation from 0.692 to 0.972 and reduces RMSE from 2.034 to 0.753 on 12 held-out configurations, providing evidence of complementary predictive information within the evaluated settings.
Critical weaknesses
-
The independent contribution of power-mean alignment is not established against strong baselines.
- On MS MARCO (Table 1), PowerJEPA without hard negatives (54.31) is 1.65 points below InfoNCE (55.96). With hard negatives, PowerJEPA (55.98) remains below InfoNCE + hard negatives (56.72) and is only 0.12 points above TERM + SIGReg + hard negatives (55.86).
- The comparison between calibrated PowerJEPA (56.81) and uncalibrated InfoNCE (56.72) does not isolate the training objective from post-hoc spectral calibration. Appendix B.5 states that the calibration study uses its own inference implementation and 37-task aggregation, that "its absolute means should not be compared directly with ... the main table's 39-task aggregation," and that "these results do not establish superiority over calibrated InfoNCE."
- On AllNLI (Table 2), the 0.25-point average advantage over InfoNCE (75.49 vs. 75.24) is driven largely by a single task, STS22.v2 (+9.73). The image results are similarly small (e.g., +0.13 CMC@5 in Table 4), and no variance is reported.
-
The incremental contribution of the capacity–shape description requires clearer positioning against prior work.
$\alpha $-ReQ [1] already investigates representation quality through eigenspectrum decay and explicitly acknowledges that a favorable decay coefficient alone is not sufficient for good downstream performance. Although the paper includes an ED + single slope baseline (Table 5), it neither cites $\alpha $-ReQ nor specifies how this baseline relates to it, leaving the incremental contribution of combining effective dimension with spectral curvature insufficiently contextualized.
[1] Agrawal K K, Mondal A K, Ghosh A, et al. $\alpha $-ReQ: Assessing Representation Quality in Self-Supervised Learning by measuring eigenspectrum decay. Advances in Neural Information Processing Systems, 2022, 35: 17626-17638.
Questions for authors
Clarification requests and targeted validation
-
Independent contribution of power-mean alignment: Which training configurations and hard-negative supervision settings are shared across PowerJEPA, InfoNCE, and TERM, and how are their tuning budgets matched?
-
Equivalent post-hoc spectral calibration: How does InfoNCE perform when given the same post-hoc spectral calibration procedure, access to calibration data, and development-based selection budget as PowerJEPA, evaluated on the same task set with the same inference implementation?
-
Relationship to $\alpha $-ReQ: Does ED + single slope use the decay-coefficient estimation procedure of $\alpha $-ReQ [1]? Please clarify their methodological relationship and explain what additional information the capacity–shape description captures beyond this baseline.
Minor correction
On line 414, "PowerJEPA consistently outperforms Lepanto" should read "PowerJEPA consistently outperforms LeJEPA".
3 comments
[W2. Positioning against -ReQ]
Thank you for pointing out this relevant work. We cited -ReQ and clarify that spectral decay as a representation-quality diagnostic is an established idea. Our incremental contribution is to examine whether head-to-tail spectral curvature provides additional predictive information beyond effective dimension and a single decay slope, supported by the comparisons in Table 5. We have revised Section 2 to cite -ReQ and Section 6.4 to clarify our focus on the additional predictive value of curvature beyond effective dimension and a single decay slope.
[Q1. Shared training configurations, hard negatives, and tuning budgets]
All methods share the same network architecture and common experimental configuration. Learning rate and batch size are tuned independently for each method, and we report the best-performing configuration found in each search. In the explicit-hard-negative comparisons, all methods receive hard-negative supervision; InfoNCE additionally uses in-batch negatives. For example, the contrastive learning results across different learning rates and batch sizes are shown below.
| Batch setting | LR | Hard negatives | Cls. | Clust. | Pair | Rerank | Retr. | STS | Sum. | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| 8192 | 2 × 10⁻⁵ | 1 | 65.21 | 43.52 | 74.57 | 42.38 | 37.48 | 71.13 | 26.35 | 54.28 |
| 1024 | 2 × 10⁻⁵ | 1 | 65.52 | 44.01 | 75.65 | 43.70 | 43.43 | 73.30 | 24.47 | 56.41 |
| 256 | 2 × 10⁻⁵ | 1 | 65.34 | 42.79 | 73.69 | 43.37 | 40.94 | 71.94 | 24.98 | 55.09 |
| 128 | 2 × 10⁻⁵ | 1 | 65.28 | 42.12 | 73.16 | 43.04 | 39.80 | 71.37 | 25.80 | 54.51 |
| 256 | 1 × 10⁻⁵ | No | 63.78 | 44.81 | 73.10 | 43.22 | 41.78 | 71.96 | 22.97 | 55.32 |
| 256 | 2 × 10⁻⁵ | No | 65.22 | 44.01 | 73.28 | 41.65 | 34.11 | 70.14 | 27.66 | 53.27 |
| 8192 | 2 × 10⁻⁵ | No | 64.12 | 44.06 | 74.13 | 42.73 | 39.33 | 71.50 | 26.39 | 54.69 |
[Q3. Relationship between the single-slope baseline and -ReQ]
A single slope summarizes spectral decay over a fitted interval. Our curvature statistic captures the difference between decay slopes over ranks 8–128 and 129–512, describing head-to-tail variation that a single fitted slope does not explicitly represent. Under the same prediction protocol, replacing the single-slope descriptor with curvature alongside ED improves Spearman correlation from 0.902 to 0.972 and reduces RMSE from 1.570 to 0.753. This supports the additional predictive value of curvature within the evaluated configurations.
[Minor correction]
Thank you for identifying the typo. We replaced “Lepanto” with “LeJEPA”.
Thank you for the additional experiments. The matched calibration and three-seed results make the paper's positioning clearer, but they also show that PowerJEPA still trails InfoNCE. I find the contribution relatively limited, and I maintain my rating.
Thank you very much for your time and feedback, which have helped us greatly improve the quality of our paper.
We have uploaded a revised version of the paper, with all changes highlighted in blue. All references to pages, sections, figures, tables, and equations use the revised version's numbering.
[W1.1 Independent contribution and performance against strong baselines]
Our main contribution is to adapt the JEPA framework to text representation learning and address the limitations of its original formulation in this setting. The relevant LeJEPA comparisons show improvements of 4.61 points without explicit hard negatives and 2.48 points with hard negatives on MS MARCO. PowerJEPA also improves over TERM + SIGReg by 1.37 points on AllNLI.
[W1.2 & Q2 Post-hoc calibration] The table below reports results with the same post-hoc calibration applied to InfoNCE and TERM + SIGReg. Our main contribution is to adapt the JEPA framework to text representation learning and address the limitations of its original formulation in this setting. We do not claim uniform superiority over strong text-embedding baselines such as contrastive learning. We have updated the corresponding discussion in the main text to report these matched-calibration results (page 9 of the revised paper).
[W1.3 Multiple-seed evaluations]
We supplemented the original seed-42 experiments with seeds 43 and 44. The tables below report the mean ± standard deviation across all three seeds. We have updated Tables 1 and 2 in the main text to report results averaged over three seeds (P. 7 of the revised paper).
MS MARCO
AllNLI