- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
ProteinJEPA adds latent prediction to MLM or next-token prediction. The main recipe predicts intermediate representations from an EMA teacher. Experiments show improvements from continued pretraining, particularly on structural retrieval and remote homology.
Critical strengths
- Useful experimental controls. The paper includes multiple pretraining seeds, step- and compute-matched comparisons, and negative results.
- Consistent gains on some tasks. Structural retrieval and remote homology improve across the main ESM2 experiments.
Critical weaknesses
- The new insight is unclear. Applying JEPA to proteins is worth testing, but the paper does not clearly explain what changes in the representations or why particular biological tasks improve. Since the contribution is mainly experimental, the reported gains need to establish a stronger practical benefit or clearer new understanding.
- Simpler alternatives are not sufficiently tested. Comparisons with original ESM2 intermediate-layer embeddings, different single target layers, and simpler distillation objectives would clarify whether the full recipe is needed.
- The compute advantage remains uncertain. Section 4.2 gives MLM-only extra steps at 1% of the peak learning rate. This matches the additional compute, but does not establish whether the gains persist with a schedule tuned for the full budget.
- The wall-clock comparison changes several factors. It uses a different target and regularizer from the controlled study. Table 14 also assigns pretrained ESM2 learning rates of 3×10⁻⁴ for JEPA and 5×10⁻⁴ for MLM-only. These comparisons do not isolate the effect of the objective.
- The practical scope is narrow. ESM2 experiments stop at 150M despite the claimed size trend, and AMPLIFY results are mixed. ESM2-650M and structure-aware models such as SaProt, ProSST, and the ProstT5 encoder would help establish practical relevance, especially on the structure-related tasks where gains are largest. Their different input requirements should be made explicit.
- The presentation needs clearer priorities. The abstract emphasizes numbers over the problem and main finding. The two training recipes are difficult to track. Table 10 has overlapping introductory descriptions and very small text. These are secondary concerns, but they make the evidence harder to assess.
Questions for authors
- Do the gains hold with equally tuned MLM schedules and matched learning rates?
- Do intermediate-layer extraction or simpler distillation methods give similar gains? What direct evidence explains the representation changes?
- Does the same recipe work on ESM2-650M, and how competitive is it with structure-aware representations under clearly specified input conditions?
6 comments
(Continued)
"The wall-clock comparison changes several factors"
Agreed: it does not isolate the objective. It is a secondary study ("The controlled study ... is the primary evidence", Sec. 4); the headline results (Table 2, Fig. 1) do not use it. "AMPLIFY results are mixed" refers to it ("near parity (6/5/8)" Sec. 5.5).
"The presentation needs clearer priorities"
The abstract has been revised, and Table 1 and the opening of Sec. 5 say which study each results section uses. We trimmed descriptions from the appendix and updated the font size of Table 10.
I thank the authors for their time and their response. I will raise my score to 2 to encourage the authors' future research. While the paper still fails to address my primary concerns, I believe it could be suitable for publication at a different conference—though not ICLR.
- Applying JEPA does not constitute innovation, it is merely following a trend, prioritizing speed of execution over scientific discovery.
- The experiments were conducted using ESM-C rather than the models I suggested (such as ESM2-650M, ProSST, or SaProt), which is quite puzzling—wouldn't it have been easier to run experiments using ESM2-650M?
Thank you for the specific requests. You ask for "a stronger practical benefit or clearer new understanding". The revision has both: per-layer probes that show what the latent loss changes, and ESM-C 600M trained from scratch, where MLM+JEPA beats compute-matched MLM-only on 18 of 19 tasks (18/1/0, p<0.001, one seed). In other words, the models are better for the same cost both in from scratch and continued training.
Counts are wins/losses/ties (W/L/T) over the 19 tasks for the first-named model; p is the one-sided sign test of Table 2. Appendix B and Tables 15 to 23 are in the revised paper.
We ran ESM-C 600M instead of ESM2-650M, as it is a newer model of similar size and claims superior importance (and would likely be of more interest to users). We did not run the structural token models: the paper tests a training objective with a protein sequence model, data and compute held equal. SaProt and ProSST take structure tokens as input, and the ProstT5 encoder reads sequence alone but was trained with them, so comparing with them measures the value of structure data, not the objective.
"The practical scope is narrow" (and Q3)
Trained from scratch (Appendix A.3), ESM-C 600M with MLM+JEPA also beats step-matched MLM-only (16/2/1, p=0.001). Against compute-matched MLM-only, SCOPe-40 is 0.439 vs 0.351, remote homology 0.658 vs 0.600, and the median gain +0.042 (Tables 2 and 19). ProGen3-112m also wins from scratch (10/3/3 of 16 tasks, p=0.046; Sec. 5.6). Continued pretraining of the released ESM-C 600M gains on the same two tasks as ESM2: SCOPe-40 +0.043 and remote homology +0.032 against MLM-only given 17% more steps (10/7/2, p=0.31; Appendix B.4).
Q1. "Do the gains hold with equally tuned MLM schedules and matched learning rates?"
Yes. In the controlled study the learning rates are matched: "Within each pair, optimizer, schedule, learning rate, batch, masking rate, and data seed are matched" (Sec. 4.2). The gain also holds against each new MLM-only schedule (continued; Table 18).
"what changes in the representations or why particular biological tasks improve" (and Q2)
In continued pretraining, the last layer becomes a better readout of fold information the model already holds. We probed every layer of off-the-shelf ESM2 (8M to 650M) and of the MLM-only and MLM+JEPA models (35M and 150M, three seeds) on SCOPe-40 and remote homology (Appendix B.1).
That is possibly why the fold tasks gain most, and why their gain grows with size.At 600M from scratch MLM+JEPA leads at every layer on SCOPe (Appendix B.4). Alternatively, it could be due to structural tasks (e.g 2d, 3d, folds) improving with pretraining and scale monotonically, while other tasks do not, as has been shown in other, cited, works.
"Simpler alternatives are not sufficiently tested" (and Q2)
The obvious alternatives, continued language modelling (MLM and generative/NTP) and the off-the-shelf model, are already the paper's baselines, and MLM+JEPA beats both. We also ran your three suggestions. Simpler distillation and other target layers are in the table (ESM2-35M, continued; Appendix B.3); the original intermediate layers are below it (Appendix B.2). The frozen teacher and plain distillation match the full recipe. A last-layer target does worse.
From scratch there is nothing to distil, so the EMA teacher supplies the target.
The off-the-shelf model's best fold layer is not a substitute. It beats MLM+JEPA's last layer on SCOPe-40, by 0.046 (35M) and 0.056 (150M). Across the 19 tasks MLM+JEPA's last layer beats that layer (35M: 12/3/4, p=0.018; 150M: 13/5/1, p=0.048; Table 17) and ties the off-the-shelf layer chosen per task on held-out data (9/7/3, p=0.40; 35M), with one readout and no layer search.