acceptodds
Under review as a conference paper at ICLR 2027

ProteinJEPA: Latent prediction improves protein language model pretraining

Abstract

Protein language models are trained primarily with masked language modeling (MLM), which predicts masked amino-acid identities. Joint-embedding predictive architectures (JEPA) instead predict latent representations, but have not been applied to proteins. ProteinJEPA supplements MLM with a cosine loss for predicting the half-depth hidden states of a teacher given the unmasked sequence. On 19 tasks, with ESM2 at 35M and 150M parameters and three pretraining seeds, MLM+JEPA outperforms compute-matched and step-matched MLM-only continued training in 78 and 76 of 114 comparisons (14 losses, 22 ties). The median compute-matched gain is on structure- and homology-sensitive tasks versus elsewhere, led by SCOPe-40 retrieval and remote homology with improvements of 6.1 percentage points in Recall@1 and 2.7 points in accuracy, respectively. Gains on these tasks increase with model size from 8M to 150M. Against the off-the-shelf checkpoint, MLM+JEPA wins 81 of 114 comparisons (median ) without improving MLM loss. In random initialization the gain is smaller and replicates inconsistently across seeds (). The same recipe improves the causal ProGen3 model, beating a compute-matched next-token-prediction control on 12 of 16 tasks. Ablations show that cosine loss beats mean squared error, while adding shallower targets removes most of the task gain. JEPA-only training collapses downstream performance: latent prediction complements MLM rather than replacing it. Code: https://anonymous.4open.science/r/protJepa-FF24

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein language model, Bioinformatics, self-supervised learning, representation learning, NLP, JEPA, MLM, continued pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LcXkOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

ProteinJEPA adds latent prediction to MLM or next-token prediction. The main recipe predicts intermediate representations from an EMA teacher. Experiments show improvements from continued pretraining, particularly on structural retrieval and remote homology.

Critical strengths
  1. Useful experimental controls. The paper includes multiple pretraining seeds, step- and compute-matched comparisons, and negative results.
  2. Consistent gains on some tasks. Structural retrieval and remote homology improve across the main ESM2 experiments.
Critical weaknesses
  1. The new insight is unclear. Applying JEPA to proteins is worth testing, but the paper does not clearly explain what changes in the representations or why particular biological tasks improve. Since the contribution is mainly experimental, the reported gains need to establish a stronger practical benefit or clearer new understanding.
  2. Simpler alternatives are not sufficiently tested. Comparisons with original ESM2 intermediate-layer embeddings, different single target layers, and simpler distillation objectives would clarify whether the full recipe is needed.
  3. The compute advantage remains uncertain. Section 4.2 gives MLM-only extra steps at 1% of the peak learning rate. This matches the additional compute, but does not establish whether the gains persist with a schedule tuned for the full budget.
  4. The wall-clock comparison changes several factors. It uses a different target and regularizer from the controlled study. Table 14 also assigns pretrained ESM2 learning rates of 3×10⁻⁴ for JEPA and 5×10⁻⁴ for MLM-only. These comparisons do not isolate the effect of the objective.
  5. The practical scope is narrow. ESM2 experiments stop at 150M despite the claimed size trend, and AMPLIFY results are mixed. ESM2-650M and structure-aware models such as SaProt, ProSST, and the ProstT5 encoder would help establish practical relevance, especially on the structure-related tasks where gains are largest. Their different input requirements should be made explicit.
  6. The presentation needs clearer priorities. The abstract emphasizes numbers over the problem and main finding. The two training recipes are difficult to track. Table 10 has overlapping introductory descriptions and very small text. These are secondary concerns, but they make the evidence harder to assess.
Questions for authors
  1. Do the gains hold with equally tuned MLM schedules and matched learning rates?
  2. Do intermediate-layer extraction or simpler distillation methods give similar gains? What direct evidence explains the representation changes?
  3. Does the same recipe work on ESM2-650M, and how competitive is it with structure-aware representations under clearly specified input conditions?
6 comments

Thank you for the specific requests. You ask for "a stronger practical benefit or clearer new understanding". The revision has both: per-layer probes that show what the latent loss changes, and ESM-C 600M trained from scratch, where MLM+JEPA beats compute-matched MLM-only on 18 of 19 tasks (18/1/0, p<0.001, one seed). In other words, the models are better for the same cost both in from scratch and continued training.

Counts are wins/losses/ties (W/L/T) over the 19 tasks for the first-named model; p is the one-sided sign test of Table 2. Appendix B and Tables 15 to 23 are in the revised paper.

We ran ESM-C 600M instead of ESM2-650M, as it is a newer model of similar size and claims superior importance (and would likely be of more interest to users). We did not run the structural token models: the paper tests a training objective with a protein sequence model, data and compute held equal. SaProt and ProSST take structure tokens as input, and the ProstT5 encoder reads sequence alone but was trained with them, so comparing with them measures the value of structure data, not the objective.

"The practical scope is narrow" (and Q3)

Trained from scratch (Appendix A.3), ESM-C 600M with MLM+JEPA also beats step-matched MLM-only (16/2/1, p=0.001). Against compute-matched MLM-only, SCOPe-40 is 0.439 vs 0.351, remote homology 0.658 vs 0.600, and the median gain +0.042 (Tables 2 and 19). ProGen3-112m also wins from scratch (10/3/3 of 16 tasks, p=0.046; Sec. 5.6). Continued pretraining of the released ESM-C 600M gains on the same two tasks as ESM2: SCOPe-40 +0.043 and remote homology +0.032 against MLM-only given 17% more steps (10/7/2, p=0.31; Appendix B.4).

Q1. "Do the gains hold with equally tuned MLM schedules and matched learning rates?"

Yes. In the controlled study the learning rates are matched: "Within each pair, optimizer, schedule, learning rate, batch, masking rate, and data seed are matched" (Sec. 4.2). The gain also holds against each new MLM-only schedule (continued; Table 18).

MLM-only controlMLM+JEPA vs control, W/L/Tp
35M: one cosine schedule over all 78,390 steps13/3/30.011
35M: 17% extra steps after a new warmup to peak rate13/1/50.001
150M: extra steps after a new warmup14/4/10.015
35M: peak learning rate four times lower11/3/50.029

"what changes in the representations or why particular biological tasks improve" (and Q2)

In continued pretraining, the last layer becomes a better readout of fold information the model already holds. We probed every layer of off-the-shelf ESM2 (8M to 650M) and of the MLM-only and MLM+JEPA models (35M and 150M, three seeds) on SCOPe-40 and remote homology (Appendix B.1).

  • Off-the-shelf ESM2 loses SCOPe-40 Recall@1 between its best layer and its last, and the drop grows with size: 0.062 (8M), 0.075 (35M), 0.129 (150M), 0.145 (650M).
  • At 150M continued MLM widens that drop (0.154) and MLM+JEPA halves it (0.065). Below the top three layers the two models differ by at most 0.008 on SCOPe-40, so the change is at the top of the network.

That is possibly why the fold tasks gain most, and why their gain grows with size.At 600M from scratch MLM+JEPA leads at every layer on SCOPe (Appendix B.4). Alternatively, it could be due to structural tasks (e.g 2d, 3d, folds) improving with pretraining and scale monotonically, while other tasks do not, as has been shown in other, cited, works.

"Simpler alternatives are not sufficiently tested" (and Q2)

The obvious alternatives, continued language modelling (MLM and generative/NTP) and the off-the-shelf model, are already the paper's baselines, and MLM+JEPA beats both. We also ran your three suggestions. Simpler distillation and other target layers are in the table (ESM2-35M, continued; Appendix B.3); the original intermediate layers are below it (Appendix B.2). The frozen teacher and plain distillation match the full recipe. A last-layer target does worse.

VariantVariant vs MLM-onlyMLM+JEPA vs variant
Frozen off-the-shelf teacher instead of EMA14/2/3, p=0.0024/3/12, p=0.50
Plain distillation from that teacher: no predictor, no regularizer14/3/2, p=0.0066/7/6, p=0.71
Target at layer 9 of 12, not layer 610/4/5, p=0.0907/5/7, p=0.39
Target at the last layer (12 of 12)11/4/4, p=0.05913/3/3, p=0.011

From scratch there is nothing to distil, so the EMA teacher supplies the target.

The off-the-shelf model's best fold layer is not a substitute. It beats MLM+JEPA's last layer on SCOPe-40, by 0.046 (35M) and 0.056 (150M). Across the 19 tasks MLM+JEPA's last layer beats that layer (35M: 12/3/4, p=0.018; 150M: 13/5/1, p=0.048; Table 17) and ties the off-the-shelf layer chosen per task on held-out data (9/7/3, p=0.40; 35M), with one readout and no layer search.

AuthorsOpenReview ↗

(Continued)

"The wall-clock comparison changes several factors"

Agreed: it does not isolate the objective. It is a secondary study ("The controlled study ... is the primary evidence", Sec. 4); the headline results (Table 2, Fig. 1) do not use it. "AMPLIFY results are mixed" refers to it ("near parity (6/5/8)" Sec. 5.5).

"The presentation needs clearer priorities"

The abstract has been revised, and Table 1 and the opening of Sec. 5 say which study each results section uses. We trimmed descriptions from the appendix and updated the font size of Table 10.

I thank the authors for their time and their response. I will raise my score to 2 to encourage the authors' future research. While the paper still fails to address my primary concerns, I believe it could be suitable for publication at a different conference—though not ICLR.

  1. Applying JEPA does not constitute innovation, it is merely following a trend, prioritizing speed of execution over scientific discovery.
  2. The experiments were conducted using ESM-C rather than the models I suggested (such as ESM2-650M, ProSST, or SaProt), which is quite puzzling—wouldn't it have been easier to run experiments using ESM2-650M?
Reviewer_H7tJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adds a latent-prediction (JEPA) loss to masked language modeling for protein LMs. The student model's last layer is trained to predict the middle-layer representation of a slowly updated teacher copy. Experiments cover ESM2 (8M-150M), from both pretrained checkpoints and random initialization, plus ProGen3 with next-token prediction. The paper reports that MLM+JEPA beats step- and compute-matched MLM-only, with the largest gains on fold and homology tasks.

The experiments are careful, but the paper claims a general improvement to pretraining while its results show a narrow benefit: limited to continued training, concentrated in two tasks, and possibly explained by copying the pretrained model's middle-layer information.

Critical strengths

The paper is readable, the comparisons are paired across seeds, and the authors report negative results (from-scratch training, JEPA-only) openly rather than hiding them.

Critical weaknesses
  1. The paper claims much more than its results support. The title says latent prediction "Improves Protein Language Model Pretraining," but the results are much narrower: a) JEPA alone does not work. Under the main recipe, JEPA-only loses against MLM-only. b) From random initialization, which is what pretraining normally means, there is no reliable gain. The paper states that "MLM+JEPA does not produce a reliable improvement across the 19 tasks from random weights" (Sec. 5.3). The meaningful gains are concentrated in two tasks: SCOPe-40 (+0.0614) and remote homology (+0.0267). Elsewhere the effect is small; for example, "subcellular localization does not change (+0.0008)" (Sec. 5.2). The title and abstract should be reduced in scope to continued pretraining.

  2. I'm not sure when anyone would use this in practice. The method only helps when continuing to train an existing checkpoint, but continued training itself doesn't seem to help. Against simply using the off-the-shelf checkpoint, the median gain is +0.0068 (Table 2).

  3. The gain may come from copying what the pretrained model already knows, not from learning anything new. As I understand the setup:

    • The target is the teacher's middle layer, chosen because "Intermediate layers of protein language models can carry more structure and homology information than the final layer" (Sec. 3).
    • In continued training, the teacher is a slowly updated copy of the original checkpoint.
    • All evaluation reads the last layer: "linear probes (logistic or ridge regression on frozen final-layer embeddings of the trained encoder)" (Sec. 4.2). So the JEPA loss asks the last layer to reproduce what the original model's middle layer already contains, and the probes only look at the last layer. A simpler explanation for the results is that JEPA moves existing structural information into the layer being tested, rather than learning something new. This fits the two main negative results: there is no gain from scratch, where there is no good middle layer to copy, and JEPA by itself collapses, because copying without MLM gives the model nothing to learn from. The authors acknowledge this gap themselves: "direct layer-wise measurements are needed to establish that mechanism" (Sec. 6).
Questions for authors
  • How does the original checkpoint's middle layer score on SCOPe-40 and remote homology, compared with MLM+JEPA's last layer?
  • Does MLM plus distillation from a frozen copy of the original model match MLM+JEPA?
  • Run the same probes on the original checkpoint's middle layer. If it already scores near MLM+JEPA's last layer, is it the case that JEPA has simply relocated information?
  • In what practical setting would one continue-pretrain a protein LM with this method rather than use the off-the-shelf checkpoint?
4 comments

Thank you for the careful reading and actionable hypotheses. We ran the experiments you asked for.

Trained from scratch, where "there is no good middle layer to copy", ESM-C 600M with MLM+JEPA beats compute-matched MLM-only on 18 of 19 tasks (18/1/0, p<0.001, one seed), and at every layer on SCOPe-40. ProGen3-112m also gains from scratch (Sec. 5.6). For continued pretraining of smaller ESM2 on SCOPe-40 and remote homology, the last layer does indeed learn a representation that seemingly also holds structural information from the middle layers.

Counts are wins/losses/ties over the 19 tasks for the first-named model; p is the one-sided sign test of Table 2. Appendix B and Tables 15 to 23 are in the revised paper.

"The paper claims much more than its results support"

To bring the claims in line with the results, we changed the title to "ProteinJEPA: Latent prediction complements protein language model training". We also now states that in continued pretraining the fold gain is at the last layer, not the best layer.

We did not limit the scope to continued pretraining, as (b), "From random initialization ... there is no reliable gain", holds only for ESM2 at 35M and 150M in the controlled study (Sec. 5.3). Two other models (ESMC, Progen3) gain from scratch. ESM-C 600M is new since submission (Appendix A.3): MLM+JEPA also beats step-matched MLM-only (16/2/1, p=0.001; Tables 2 and 19), and the abstract now states its gain. ProGen3-112m with next-token prediction was already in the paper: "From random initialization, NTP+JEPA scores 10/3/3 against either control" (16 tasks, p=0.046; Sec. 5.6).

"JEPA moves existing structural information into the layer being tested" (and Q1, Q3)

On SCOPe-40 and remote homology in continued pretraining, yes. The off-the-shelf model's half-depth layer scores higher than MLM+JEPA's last layer on both (Table 20), and the three models' best layers are within about 0.015 (Table 16).

LayerSCOPe-40, 35MSCOPe-40, 150MRH, 35MRH, 150M
Off-the-shelf, last0.3830.4240.6460.687
Off-the-shelf, half depth (the target)0.4230.5410.6790.735
MLM-only, last0.3740.4030.6480.686
MLM+JEPA, last0.4130.4970.6640.719

RH is remote homology; trained models are medians of three seeds.

Off-the-shelf ESM2 loses 0.062 SCOPe-40 Recall@1 between its best layer and its last at 8M, and 0.145 at 650M. Continued MLM does not correct it: at 150M the drop is 0.129 off the shelf, 0.154 after MLM-only and 0.065 after MLM+JEPA (Table 15).

The best fold layer is not a substitute. Over the 19 tasks MLM+JEPA's last layer beats the off-the-shelf model's best SCOPe-40 layer (12/3/4, p=0.018 at 35M; 13/5/1, p=0.048 at 150M; Table 17). It ties the off-the-shelf layer chosen separately for each task on held-out data (9/7/3, p=0.40; 35M), with one output and no layer search.

At 600M from scratch, MLM+JEPA is ahead of compute-matched MLM-only at all 36 layers on SCOPe-40, and its best layer on the full test set scores 0.466 against 0.391 (Table 22).

"I'm not sure when anyone would use this in practice" (and Q4)

To train a better protein language model at the same cost. At 600M from scratch, 17% more compute spent on the latent loss gives the gains above, and the same 17% spent on more MLM steps gives no reliable gain (compute-matched against step-matched MLM-only: 10/8/1, p=0.41; Table 21).

A second use is training an existing model. Continued MLM alone barely helped here: against the off-the-shelf checkpoint "MLM-only continuation is 52/35/27", with a median of +0.0012 (Sec. 5.1). MLM+JEPA is 81/20/13 with the median +0.0068 you quote, and at 150M it gains 0.073 on SCOPe-40 and 0.032 on remote homology (medians of three seeds, Table 20). On ProGen3, "Both arms improve over the checkpoint, with median gains of +0.017 for NTP+JEPA and +0.007 for NTP-only" (Sec. 5.6). This is important in a number of research works, such as adding new datasources (e.g. genomics, clinical proteomics) or experiments (e.g. non standard amino acids), research or improving undertrained existing models.

AuthorsOpenReview ↗

(Continued)

Q2. "Does MLM plus distillation from a frozen copy of the original model match MLM+JEPA?"

No detectable difference. It is our loss with the teacher frozen and the predictor and regularizer removed. It beats MLM-only 14/3/2 (p=0.006), and MLM+JEPA is 6/7/6 against it (p=0.71; ESM2-35M, Table 18). So predicting the teacher's representations is what matters.

"concentrated in two tasks"

In continued ESM2 while the fold tasks have the largest gains, gain is not confined to them: "The next-largest gains are outside the subset: peptide-HLA binding (+0.0169), signal-peptide residues (+0.0137), fluorescence (+0.0124), and β-lactamase (+0.0117), each positive in at least five of six runs" (Sec. 5.2). At 600M gain is broad, with a median of +0.042 over the 19 tasks. Subcellular localization, which "does not change (+0.0008)" in continued ESM2, gains 0.071, and fluorescence, disorder and EC classification gain 0.04 to 0.06 (Table 19).

Reviewer_H7tJOpenReview ↗

Remain unconvinced

Thanks for your response. Overall, this doesn't change my assessment that this is a modest technical advance of unclear relevance (the latter is more important to me). I'll retain my score.

We have shown the model improves both in from scratch and continued, and at larger scales, better than the existing baseline and SOTA (MLM and causal NTP). We also showed this improves multiple tasks, not just structure and that continued language modelling failed at that in many cases.

Reviewer_MdZWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper investigates adding a JEPA-style latent prediction loss to PLM pretraining alongside MLM loss.
  2. They tested it on ESM2-35M and 150M (three seeds, + 19 tasks), both continued from pretrained checkpoints and from scratch (against step-matched, compute-matched and off-the-shelf baselines).
  3. In continued pretraining MLM+JEPA beats the compute-matched control (in 78 of 114 comparisons) with the biggest gains on SCOPe-40 retrieval and remote homology. From scratch they report no reliable gain.
  4. They then show that the recipe transfers to the causal ProGen3 model.
  5. They then perform several ablation studies and show that cosine beats MSE, adding shallower targets removes most of the structural gain and JEPA-only training collapses downstream performance
  6. An additional wall-clock study across five backbones gives mixed results.
Critical strengths
  1. Gains seem real. a. Core improvements on SCOPe retrieval and remote homology seem real and are substantial enough to be interesting. b. Despite it being limited to continued pretraining they are still non-trivial gains.

  2. Thorough methodology a. The methodology is pretty thorough and defendable (independent pretraining seeds, comparisons against both continued MLM and the original checkpoint, and the checks on alternative metrics and aggregation choices). b. They also show the results survives several ways of summarising the experiments, which makes it more convincing than a favourable average from one run. c. Ablation studies are well motivated and informative

Critical weaknesses
  1. Compute-matched baseline could be stronger a. The MLM control gets an extra 11,390 steps only after completing the original cosine schedule, at just 1% of the peak learning rate? This seems low. It matches the measured compute budget but doesn’t establish how well MLM would perform with a schedule designed for the full budget. b. The wall-clock study provides supporting evidence, but its pretrained ESM2 comparisons also use different learning rates for MLM and MLM+JEPA? It’s not clear how these were chosen or what the impact of this choice is. c. Overall I appreciate the budget matching but would like to see an MLM schedule designed for the full budget and possibly some clarification of how the wall-clock learning rates were chosen? This would convince me more that JEPA is a better use of the additional compute.

  2. The source of the improvement isn’t fully isolated. a. The EMA teacher and VISReg were adopted together and never ablated separately , so without an MLM+VISReg-only control it’s unclear to me how much of the gain specifically requires predicting the teacher representations. b. Also the paper uses frozen final-layer embeddings but draw targets from intermediate layers?
    c. I’d like to see some results from validation-selected intermediate layers of the original and MLM-only models. This would help establish whether the method improves the available representations or mainly just makes existing information easier to access at the final layer.

Questions for authors

How were the LR/schedulers selected and did all the methods receive comparable tuning.

1 comment

Thank you for the careful review, and for noting our gains and methodology. We ran the controls you asked for: MLM+JEPA still beats MLM-only on a schedule designed for the full budget, and the gain comes from predicting the teacher's representations. We also trained a larger ESM-C 600M from scratch: MLM+JEPA beats compute-matched MLM-only on 18 of 19 tasks (18/1/0, p<0.001, one seed), so the gain is no longer limited to continued pretraining.

Counts are wins/losses/ties (W/L/T) over the 19 tasks for the first-named model; p is the one-sided sign test of Table 2. Appendix B and Tables 15 to 23 are in the revised paper.

"would like to see an MLM schedule designed for the full budget"

We ran three. MLM+JEPA wins against each (Table 18), and none of these controls reliably beats step-matched MLM-only (sign p from 0.39 to 0.97; Table 21).

MLM-only controlMLM+JEPA vs control, W/L/TpSCOPe-40 gain
35M: one cosine schedule over all 78,390 steps13/3/30.011+0.042
35M: the 17% extra steps after a new warmup to peak rate13/1/50.001+0.044
150M: extra steps after a new warmup14/4/10.015+0.088

"How were the LR/schedulers selected and did all the methods receive comparable tuning"

In the controlled study the two arms share optimizer settings. We searched neither arm's learning rate or schedule: "Within each pair, optimizer, schedule, learning rate, batch, masking rate, and data seed are matched" (Sec. 4.2). The peak rates are 1e-3 from scratch, following "Cramming Protein Language Model Training in 24 GPU Hours" (Frey et al.) and "Cramming: Training a Language Model on a Single GPU in One Day" (Geiping & Goldstein), and 2e-4 in trained model continuation. MLM+JEPA's loss weight is fixed at 1, and the weight of its regularizer (VISReg) was set in short pilot runs. MLM+JEPA also beats MLM-only run at a four times lower peak rate (5e-5, 35M: 11/3/5, p=0.029).

On the wall-clock study ("its pretrained ESM2 comparisons also use different learning rates") you are right: the arms ran at 3e-4 and 5e-4 (Table 14), both carried over from earlier runs, so that study does not fully separate the objective from the rate. The paper treats it as secondary ("The controlled study ... is the primary evidence", Sec. 4), and the headline results (Table 2, Fig. 1) do not use it.

"The EMA teacher and VISReg were adopted together and never ablated separately"

The gain comes from predicting the teacher's representations (ESM2-35M, continued; Table 18):

  • MLM + VISReg, without predicting the teacher: no reliable gain over MLM-only (8/5/6, p=0.29), and MLM+JEPA beats it 12/3/4 (p=0.018).
  • A frozen teacher (the off-the-shelf checkpoint) instead of the EMA teacher: no detectable difference (MLM+JEPA 4/3/12 against it).
  • The predictor network and VISReg also removed, leaving plain distillation of the frozen half-depth state: no detectable difference (MLM+JEPA 6/7/6 against it), and it beats MLM-only 14/3/2 (p=0.006).

From scratch there is no pretrained model to distil from, so the teacher supplies the target.

"improves the available representations or mainly just makes existing information easier to access at the final layer"

First, on why we "draw targets from intermediate layers": our fundamental reason was reducing compute costs, as part of our comparing the compute budgets with different approaches, and a half-depth target costs 17% more compute than MLM-only, against about 37% for a last-layer target (Sec. 3). Secondly, intermediate layers can carry more information than the final layer (obersved heuristically in other works, we were not aware of formal work on this when we did the paper). The cheaper target is also the better one: half depth beats a last-layer target (13/3/3, p=0.011) and ties a layer-9 target (7/5/7; Table 18).

To your question: on the structural fold tasks in continued pretraining, easier access at the final layer is seemingly influential. With layers selected on held-out data, the best layers of the off-the-shelf, MLM-only and MLM+JEPA models are within about 0.015 on SCOPe-40 and remote homology (Table 16). However, the drop from best to final layer changes: at 150M on SCOPe-40, 0.129 off the shelf, 0.154 after MLM-only, 0.065 after MLM+JEPA (Table 15).

On SCOPe-40 the off-the-shelf model's best layer does beat MLM+JEPA's final layer, by 0.046 (35M) and 0.056 (150M). Across all 19 tasks, MLM+JEPA's final layer beats that layer (35M: 12/3/4, p=0.018; 150M: 13/5/1, p=0.048; Table 17). It also ties the off-the-shelf layer chosen separately for each task on held-out data (35M: 9/7/3, p=0.40). So in continued pretraining, MLM+JEPA gives one readout that needs no layer search.

From scratch, the representation itself improves: at 600M, MLM+JEPA leads compute-matched MLM-only on SCOPe at every layer (best layer chosen on held-out protein families: 0.534 vs 0.450; Table 22).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.