acceptodds
Under review as a conference paper at ICLR 2027

Towards All-Atom Protein World Models: Learning and Diagnosing Predictive Latent Representations

Abstract

All-atom protein models compress atomic structure into latent representations, often using reconstruction or generative objectives that require these latents to preserve precise coordinate information. Yet, residue properties depend on both local atomic environments and longer-range 3D interactions. Joint-Embedding Predictive Architectures (JEPAs) offer an alternative: predicting representations in latent space rather than reconstructing the raw input. In vision, this shifts learning away from pixel-level reconstruction toward higher-level features that transfer across tasks. We ask whether the same approach can help protein representations capture both residue-level information and broader structural context useful for protein structure and function tasks. Using the La-Proteina encoder, we train JEPA, JEPA+SIGReg and LeJEPA on masked protein structures and evaluate them with StructTokenBench. We study how masking, positional information, rotations and cropped views affect the learned representations. JEPA with 70% random masking scores higher than the Cα-conditioned La-Proteina baseline on 17 of 21 functional-site and flexibility scores. In JEPA+SIGReg, I-JEPA-style absolute positional embeddings create a strong residue-index shortcut. Relative positioning suppresses this direct shortcut and scores higher on 19 of 21 functional-site and flexibility outputs. LeJEPA reveals a different shortcut: its representations organise residues by chain length, motivating us to replace full-protein views with fixed-length crops. Evaluating all 12 encoder blocks shows that blocks 4-6 provide the highest score on 17 of 24 task/splits, while block 4 outperforms every final representation in our 18-model comparison on 22/24 task/splits. These results show the potential of latent prediction for all-atom protein representations while highlighting the importance of protein-specific representation design.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
protein representation learning, all-atom protein structure, joint-embedding predictive architectures, self-supervised learning, latent representations, protein world models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DkxCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates predictive latent representation learning for all-atom protein structures by adapting JEPA, JEPA+SIGReg, and LeJEPA to the La-Proteina encoder. Its main contributions are the identification of residue-index and chain-length shortcuts, protein-specific view designs intended to mitigate these shortcuts, and a layerwise analysis showing that intermediate representations often transfer substantially better than the final 8D latents. Validation uses frozen-encoder evaluation on StructTokenBench, supplemented by masking, positional encoding, cropping, rotation, and representation diagnostics. The results demonstrate useful downstream representations and informative failure modes, although performance is task-dependent and predictive objectives do not consistently outperform reconstruction baselines.

Critical strengths

Quantitative diagnosis of protein-specific shortcuts. The paper goes beyond downstream scores to identify residue-index organization in JEPA+SIGReg and chain-length organization in LeJEPA. In particular, replacing absolute positioning with relative positioning sharply reduces exact-index neighborhood purity while preserving effective rank and improving most downstream task/splits. This provides actionable evidence that apparently noncollapsed representations can nevertheless be dominated by shortcuts.

Thorough examination of where transferable information is retained. The layerwise analysis shows substantial downstream gains from intermediate representations, supported by comparisons against randomly initialized features and additional evaluation in JEPA. This is consequential because evaluating only the final latent would substantially underestimate the information learned by these encoders.

Critical weaknesses

The comparisons do not isolate the contribution of predictive pretraining. The strongest transfer results use intermediate 768D representations, but the reconstruction baselines are not subjected to the same layerwise analysis. Indeed, the fully latent autoencoder already achieves the highest average functional-site and flexibility scores among the reported 100k final-latent configurations. Without comparable intermediate-layer reconstruction baselines, it remains unclear whether the strongest results reflect a benefit specific to predictive objectives. Furthermore, the absence of sequence-only and backbone-only controls leaves the contribution of all-atom information unresolved. These omissions limit the methodological significance beyond applying existing objectives to a new input setting.

The chain-length mitigation results do not establish a clean link between shortcut removal and improved transfer. The controlled cropping comparison reduces chain-length organization but also sharply lowers effective rank. The configurations used for the main downstream results subsequently change local masking, batch size, training duration, and the eligible protein population. Consequently, their performance cannot be attributed specifically to removing the chain-length shortcut. Although these differences are disclosed, the missing controlled comparison and single-seed evaluation weaken a central practical conclusion about effective view design.

Questions for authors

Clarification: How are overlaps between the pretraining corpus and StructTokenBench test proteins assessed? Please distinguish exact-chain overlap from sequence-homology overlap. Also, are encoder layers and model configurations selected using downstream validation sets, or are the reported best-layer comparisons descriptive comparisons on test results?

Additional evidence that could change my assessment: How do the intermediate representations of the fully latent autoencoder compare with those of JEPA/LeJEPA under the same downstream protocol and comparable training budgets? This would help determine whether the strongest transfer results are specific to predictive pretraining.

Additional evidence that could change my assessment: For the final cropped configuration with 50% local masking, does chain-length organization remain low while effective rank remains high? Can its downstream performance be compared with an uncropped configuration matched for masking, batch size, eligible proteins, and training budget? Replication of this key comparison across seeds would strengthen the interpretation.

Additional evidence that could change my assessment: Are the main transfer gains retained when side-chain information is removed and only amino-acid identity and backbone coordinates are provided? This would clarify whether all-atom detail contributes materially to the reported representations.

2 comments

We are grateful to the reviewer for the clear and focused questions. We addressed a number of points in The Response to All Reviewers. Following the reviewer's suggestion, we now present the paper as a diagnostic study, as described in the Response to All Reviewers, item 1. We answer the remaining points here.

Q1. How are overlaps between the pretraining corpus and StructTokenBench test proteins assessed? Please distinguish exact-chain overlap from sequence-homology overlap. Also, are encoder layers and model configurations selected using downstream validation sets, or are the reported best-layer comparisons descriptive comparisons on test results?

The overlap is addressed in the Response to All Reviewers, item 10, and Appendix E.3. As requested, we now report exact-chain overlap and sequence-homology overlap (MMseqs2) separately. Removing exact-chain overlaps changes the comparisons against the fully latent AE by at most one task.

Encoder layers are selected on the downstream validation sets. For each model and task, we report the test score of the block with the best validation score, and probe checkpoints are also chosen on validation data (Response to All Reviewers, item 6; Section 3.4). For 94 of 96 model-task pairs, the validation-chosen block is within training-seed noise of the best test block (Appendix Table 6).

Q2. How do the intermediate representations of the fully latent autoencoder compare with those of JEPA/LeJEPA under the same downstream protocol and comparable training budgets?

As requested, we now evaluate all 12 blocks of the fully latent AE under the same downstream protocol as JEPA and LeJEPA, including LeJEPA at JEPA's training budget (Response to All Reviewers, item 6; Figure 4, Appendix Figure 9). The fully latent AE's blocks tie with a randomly initialised encoder on 11 of 12 tasks, while the blocks of every JEPA-based model score higher than the fully latent AE's on at least five tasks and lower on none. At the same training budget as JEPA, LeJEPA's blocks score higher than the fully latent AE's on six tasks and lower on none (Section 4.3, Appendix Table 5).

Q3. For the final cropped configuration with 50% local masking, does chain-length organization remain low while effective rank remains high? Can its downstream performance be compared with an uncropped configuration matched for masking, batch size, eligible proteins, and training budget? Replication of this key comparison across seeds would strengthen the interpretation.

For the final cropped models trained with batch size 96, chain-length correlation stays low (0.19 to 0.51 across the four cropped models, versus 0.97 for uncropped LeJEPA) and effective rank stays near full (7.6 to 7.9 of 8; Appendix Table 8). As requested, we also ran the matched comparison over three seeds, with the same masking, batch size, eligible proteins and steps (Response to All Reviewers, item 7). Cropping lowers the chain-length correlation from 0.85 to 0.21 but does not improve downstream performance (Section 4.2, Appendix Table 7).

Q4. Are the main transfer gains retained when side-chain information is removed and only amino-acid identity and backbone coordinates are provided?

As requested, we trained JEPA+SIGReg on backbone atoms and amino-acid identity only, and on sequence only (Response to All Reviewers, item 9; Section 4.1). Side-chain atoms contribute substantially to the gains. Trained on backbone atoms and amino-acid identity, the final latent ties with random init on all 12 tasks. Trained on sequence only, it scores higher than random init on two tasks and lower on four (Appendix Table 10). Thus, all-atom detail contributes materially to the reported representations.

We appreciate the reviewer's suggestions and hope the revised manuscript answers the reviewer's questions.

Thank you again for your review. As the discussion period closes in a couple of hours, we wanted to check whether our revision addressed your concerns. In short, the revision adds layerwise evaluation of all 12 blocks of the fully latent AE, the Cα-conditioned AE and a randomly initialised encoder (Figure 4, Appendix Table 5), the matched uncropped/cropped comparison over 3 seeds (Appendix Table 7), and backbone-only and sequence-only controls showing that side-chain atoms contribute significantly (Appendix Table 10). If there is anything else you would like us to clarify or test, please let us know and we will do our best to address it in the remaining time.

Reviewer_7cEuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts JEPA-style learning to all-atom protein structures using the La-Proteina encoder. It compares an EMA-teacher JEPA, a shared-encoder JEPA+SIGReg, and LeJEPA with protein-level or residue-level view alignment. All models are trained from scratch, and their frozen representations are evaluated on StructTokenBench. The study focuses on how masking, positional information, rotations, cropping, and encoder depth affect transfer. The most useful findings concern what the objectives encourage the models to retain. Absolute-position JEPA+SIGReg learns a strong residue-index signal, while uncropped LeJEPA organises embeddings by chain length. Relative positions and fixed-length crops reduce these signals. Intermediate encoder blocks also transfer much better than the final eight-dimensional representation. These observations make a worthwhile diagnostic study, although I am less convinced that the experiments establish an advantage for predictive pretraining over reconstruction. The fully latent autoencoder is a strong baseline, several comparisons change multiple training conditions, and the main results come from single runs.

Critical strengths
  1. A concrete diagnosis of positional shortcuts The absolute-position result is unusually clear. At 70% masking, switching to relative positions reduces cross-protein exact-index purity from 95.79% to 1.68%, while effective rank stays close to eight. Downstream performance improves on 19/24 task/splits, including CatBio fold AUROC from 0.529 to 0.733. The combination of neighbourhood statistics, rank measurements, and task results gives this argument more substance than a t-SNE plot alone (Section 4.2).
  2. View design is examined rather than assumed The chain-length analysis shows why applying a generic multi-view objective to proteins can go wrong. Fixed-length crops reduce the measured length correlation from 0.884 to 0.193. The authors also report that the initial cropped configuration loses effective rank, and adjust local masking accordingly. Reporting this failed configuration is helpful: removing an easy signal does not automatically produce a useful representation (Section 4.2).
  3. The layerwise analysis has useful controls For uncropped LeJEPA, block 4 outperforms block 12 on 22/24 task/splits, with both representations fixed at 768 dimensions. The trained block 6 also exceeds its randomly initialised counterpart on 23/24 outputs. A separate six-block model and task-supervised 8D projections provide further checks. These experiments support the practical recommendation to evaluate internal layers instead of treating the final latent as the only representation (Section 4.3; Appendix D).
  4. Rotation effects are measured directly The paper tests geometric invariance as well as downstream scores. Across 400 chains and 16 rotations per chain, rotation training increases mean cosine similarity of corresponding protein-centred residue embeddings from 0.277 to 0.765. It also improves 21/24 downstream outputs in that 10k uncropped comparison. The accompanying rank measurements help distinguish invariance from simply preserving all sources of variation (Section 4.2).
  5. A broad and candid empirical study The evaluation includes functional sites, flexibility, and remote homology, with complete tables for 18 configurations at 100k steps and four cropped models at 10k. The authors acknowledge that no objective wins throughout, that cropped runs are not compute-matched, and that task-supervised 8D projections do not establish a universal 8D representation. Appendix D also includes selected SLAE-Hidden and ESM3 reference scores where evaluation definitions match.
Critical weaknesses
  1. The advantage over reconstruction remains unclear The abstract highlights wins against the C-alpha-conditioned autoencoder, but the fully latent autoencoder has the highest average functional-site AUROC (0.689) and flexibility correlation (0.294) among the 100k models. This is a stronger comparator for representations expected to retain complete structural information. Predictive models win on selected tasks, but there is no consistent overall advantage. I would present the contribution primarily as a study of representation behaviour and make the comparison with the fully latent baseline more prominent (Table 1; Section 4.1).
  2. The cropped-model comparison changes several factors The initial crop diagnostic holds masking fixed, which is useful, but the final cropped models also change local masking from 80% to 50%, use batch size 96 rather than 16, exclude chains shorter than 96 residues, and train for 10k rather than 100k steps. Batch size is particularly relevant to a regulariser computed across proteins. These results cannot isolate how much downstream improvement comes from suppressing the length signal. Matched runs separating crop length, masking, batch size, and training budget would make the design recommendations more reliable (Sections 3.3 and 4.2; Appendix C).
  3. Statistical support is insufficient for the model rankings Appendix C states that each configuration is trained once with seed 42. Many conclusions count the number of task/split wins, including small differences, without uncertainty from pretraining or downstream-head fitting. The protein-resampling intervals for rotation invariance do not address this issue. Remote homology is also reported as a mean of batch-level macro-F1 rather than a single dataset-level score; the paper should explain the aggregation and its sensitivity to batch composition. A smaller replicated set of central comparisons would be more persuasive than additional single-run variants.
  4. The bottleneck interpretation needs matched baselines The same-width block-4 versus block-12 comparison supports a late-layer decline in uncropped LeJEPA. However, there is no corresponding layerwise evaluation of the reconstruction baselines, so it remains unclear whether predictive training produces better transferable hidden states under the same readout. Task-supervised 8D projections show that each task can select useful information in eight dimensions, but they do not separate the shared latent-width constraint from objective-specific information loss. A shared pretrained width sweep and matched intermediate-layer autoencoder results would address this gap (Section 4.3; Appendix D).
  5. The all-atom and world-model framing exceeds the evidence The model predicts missing structural representations from a static view; the experiments do not test dynamics, interventions, or conformational transitions. The title says "towards," but the world-model framing still needs a narrower operational definition. Likewise, there is no backbone-only or sequence-only control establishing the contribution of side-chain coordinates. All transfer results use StructTokenBench, training chains are restricted to 50-256 residues, and overlap between pretraining chains and downstream evaluation proteins is not documented. The 50%-identity split of the pretraining collection alone does not establish separation from those downstream test sets (Sections 3.1 and 3.4).
Questions for authors
  1. Under the same frozen-layer readout and downstream-head protocol, how do intermediate JEPA and LeJEPA representations compare with intermediate representations of both autoencoders? Please include a validation-based rule for choosing the layer.
  2. Can the uncropped/cropped comparison be repeated with matched local masking, batch size, eligible proteins, and training budget? Does reducing chain-length organisation still improve downstream performance under those controls?
  3. Can the main positional, rotation, and layerwise comparisons be repeated across independent seeds, with uncertainty for the downstream metrics? For remote homology, how does dataset-level macro-F1 compare with the reported batch-averaged score?
  4. Does a wider shared pretrained latent preserve the information lost in the final 8D representation? A width sweep would help distinguish a capacity limitation from the effect of the predictive objective.
  5. How much do side-chain coordinates contribute relative to backbone-only and sequence-only inputs, and what sequence or structural overlap exists between pretraining data and StructTokenBench evaluation proteins?
3 comments

We thank the reviewer for the detailed review and the concrete suggestions. We addressed a number of points in The Response to All Reviewers. Following the reviewer's suggestion, we now present the paper as a diagnostic study and lead with the fully latent AE comparison, as described in the Response to All Reviewers, items 1 and 2. We answer the remaining points here.

Q1. Under the same frozen-layer readout and downstream-head protocol, how do intermediate JEPA and LeJEPA representations compare with intermediate representations of both autoencoders? Please include a validation-based rule for choosing the layer.

As suggested, we now evaluate all 12 blocks of both autoencoders, of every JEPA-based model and of a randomly initialised encoder under the same frozen readout and downstream head, and choose the block on validation data for each task (Response to All Reviewers, item 6; Section 3.4, Figure 4, Appendix Figure 9). The blocks of every JEPA-based model score higher than the fully latent AE's on at least five tasks and lower on none, and the Cα-conditioned AE's blocks are the lowest on 10 of 12 tasks (Section 4.3, Appendix Table 5). For 94 of 96 model-task pairs, the validation-chosen block is within training-seed noise of the best test block (Appendix Table 6).

Q2. Can the uncropped/cropped comparison be repeated with matched local masking, batch size, eligible proteins, and training budget? Does reducing chain-length organisation still improve downstream performance under those controls?

We repeated the comparison with matched local masking, batch size, eligible proteins and training steps over three seeds (Response to All Reviewers, item 7). Under these controls, cropping lowers the chain-length correlation from 0.85 to 0.21 but does not improve downstream performance: in each seed, the cropped model scores higher on one task and lower on two to four (Section 4.2, Appendix Table 7).

Q3. Can the main positional, rotation, and layerwise comparisons be repeated across independent seeds, with uncertainty for the downstream metrics? For remote homology, how does dataset-level macro-F1 compare with the reported batch-averaged score?

We repeated the positional, rotation and crop comparisons over three training seeds and use the seed variation as the uncertainty for every comparison (Response to All Reviewers, items 4 and 5; Appendix E.2). In each seed, relative positions score higher than absolute positions on four to seven tasks and lower on at most one, and rotation improves LeJEPA on eight or nine tasks and lowers it on at most one (Appendix Table 7). Due to compute limits, the layerwise curves use one training seed per model, since each model needs probes for 12 blocks on 24 splits. The same training-seed threshold is applied to the layerwise comparisons. For remote homology, dataset-level macro-F1 is higher than the batch-averaged score for 13 of 15 model-split pairs, and the best model changes only on the superfamily split (Appendix E.5, Appendix Table 9).

Q4. Does a wider shared pretrained latent preserve the information lost in the final 8D representation? A width sweep would help distinguish a capacity limitation from the effect of the predictive objective.

We thank the reviewer for suggesting the width sweep. JEPA+SIGReg with a 16D, 32D or 64D latent ties with its 8D latent on all 12 tasks, and JEPA-70 with a 64D latent scores lower than its 8D latent on eight tasks (Response to All Reviewers, item 8; Appendix Table 10). A task-supervised 768-to-8 projection, in contrast, retains most of the block-level information (Appendix Figure 8), which suggests that the width of the latent is not the limit.

Q5. How much do side-chain coordinates contribute relative to backbone-only and sequence-only inputs, and what sequence or structural overlap exists between pretraining data and StructTokenBench evaluation proteins?

As suggested, we trained JEPA+SIGReg on backbone atoms and amino-acid identity only, and on sequence only (Response to All Reviewers, item 9; Section 4.1). Side-chain atoms contribute substantially to the gains. Trained on backbone atoms and amino-acid identity, the final latent ties with random init on all 12 tasks. Trained on sequence only, it scores higher than random init on two tasks and lower on four (Appendix Table 10).

The overlap is addressed in the Response to All Reviewers, item 10, and Appendix E.3. We report exact-chain overlap and sequence-homology overlap (MMseqs2) separately, and removing exact-chain overlaps changes the comparisons against the fully latent AE by at most one task.

We appreciate the reviewer's feedback and hope these revisions resolve the points raised.

Thank you again for your review. As the discussion period closes in a couple of hours, we wanted to check whether our revision addressed your concerns. In short, the revision adds layerwise evaluation of all 12 blocks of both autoencoders, a matched masked AE and a randomly initialised encoder (Figure 4, Appendix Table 5), the matched uncropped/cropped comparison over 3 seeds (Appendix Table 7), a latent width sweep at 16D, 32D and 64D (Appendix Table 10), and backbone-only and sequence-only controls (Section 4.1). If there is anything else you would like us to clarify or test, please let us know.

Thank you for the detailed response and additional experiments. I believe the authors have addressed most of my previous concerns. In particular, the layerwise comparison with the fully latent autoencoder, the matched cropping experiments, and the latent width ablations make the experimental analysis more complete and clarify the differences between the pretraining objectives.

I also appreciate the additional multi-seed experiments for positional encoding and rotation, as well as the more detailed analysis of pretraining and test-set overlap. These additions improve the reliability of the evaluation.

However, the layerwise comparisons still rely on a single training seed, so the uncertainty in these results has not been fully addressed. In addition, although cropping reduces chain-length organization, it does not improve downstream performance. This suggests that removing shortcuts does not necessarily lead to better transferable representations.

Overall, I appreciate the authors' efforts and believe the revision has improved the paper. However, these changes do not substantially alter my overall assessment of the work, so I will maintain my original rating.

Reviewer_C3zFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper adapts three latent-prediction SSL objectives to all-atom protein structure, all on top of the La-Proteina transformer encoder, trained from scratch on ~339k PDB chains: (i) an I-JEPA-style JEPA with EMA target encoder and absolute-position cross-attention predictor; (ii) "JEPA+SIGReg" — shared encoder, no stop-gradient, SIGReg on context/target/predictor outputs; (iii) LeJEPA with one global + four local mean-pooled views, plus cropped and residue-level variants. Frozen reps are probed on StructTokenBench against two La-Proteina autoencoder baselines. Three claims: (a) predictive objectives transfer but no model dominates — JEPA-70% beats the Cα-conditioned AE on 17/21 functional+flexibility scores; (b) generic objectives exploit protein-specific shortcuts — with absolute position queries, JEPA+SIGReg latents encode residue index almost directly, relative-position biases cut this to 1.68% while improving 19/24 scores; uncropped LeJEPA encodes chain length; rotation augmentation raises orientation invariance; (c) intermediate 768-D blocks are far more informative than the final 8-D latent, and task-supervised 768→8 projections recover most of the gap.

Critical strengths
  1. The idea seems quite interesting.
Critical weaknesses
  1. The headline comparisons live in a regime a random encoder already dominates, and the baselines aren't the right controls. Table 4: a randomly initialized 768-D block 6 beats the best trained 100k-step final-8-D model (the fully latent AE) on 20/24 task/splits (e.g. CatBio SF 91.41 vs. 83.56, BindBio SF 74.58 vs. 69.65, CatInt SF 88.95 vs. 80.08, Ept SF 69.06 vs. 52.38). So most of the 1–5 point differences between models in Table 1 are dominated by the 8-D bottleneck, not by what the objectives learn — "predictive representations transfer" is weakly supported. The abstract's headline comparison also uses the Cα-conditioned AE, which deliberately excludes Cα geometry and is handicapped by design; against the fully latent AE, JEPA-70% wins only 5/21 functional+flexibility scores by my count from Table 2. There's no masked-reconstruction baseline on the same encoder/masks/compute, so the paper can't actually attribute anything to "latent prediction vs. reconstruction" — its own motivating question. The layer-wise analysis is never run for the AEs, so the abstract's "block 4 beats every final representation" compares a 768-D intermediate layer against 8-D outputs, not against the AEs' own intermediate layers — without that, the "objective-facing bottleneck" can't be pinned on JEPA objectives specifically. And beyond a few SLAE/ESM3 numbers copied into Fig.7 "where definitions match," there's no external reference point (published StructTokenBench tokenizers, ESM-2/SaProt, GearNet-style encoders). The paper's main positive claim is thus not separable from dimensionality and baseline choice.

  2. Statistical/experimental confounds undercut most win-count claims. Everything is one seed (42), no probe variance reported. Checkpoint averaging alone moves scores a lot (Table 2: LeJEPA+SO(3) online vs. EMA — Homo family 27.60 vs. 33.63, CatInt fold 59.94 vs. 56.07; uncropped LeJEPA FlexBFactor SF 3.50 vs. 7.72) — yet AEs are scored with EMA weights and everything else with online weights. Several task/splits sit at or below chance for nearly every model (BindInt fold ≈43–53, Ept fold ≈47–62 AUROC) but still count toward the "X/24" tallies, and fold/superfamily splits of the same task aren't independent anyway. Training budgets differ wildly — JEPA/JEPA+SIGReg see ~400k chain presentations (≈1.2 epochs) at batch 4, uncropped LeJEPA sees 1.6M, cropped LeJEPA 960k crops from a dataset that drops chains under 96 residues — yet cropped models sit in the same table and get credited with "highest on all three Homo splits." The uncropped→cropped effective-rank jump (7.10→1.61) is also confounded with batch size by the authors' own account, and SIGReg over just 16 eight-dim vectors per batch is a weak normality test that probably adds instability. Finally, the rotation claim ("enforcing orientation invariance improves most scores," Sec. 5) holds for LeJEPA but not JEPA+SIGReg — rel-pos 70%+SO(3) improves only 12/24 and worsens 5/6 flexibility scores (FlexRMSF fold 28.97→18.45, FlexBFactor fold 19.82→12.46), unmentioned in the text. Net effect: once you account for noise and confounds, few of the specific design conclusions (rotations, crops, residue vs. protein alignment) are actually established.

Secondary issues: the "world model" framing in the title isn't substantiated — nothing here predicts dynamics, conformational change, or interventions, it's masked latent prediction on static structures. Novelty is modest (I-JEPA/LeJEPA applied to an existing encoder; relative-position bias is standard). Pretraining and StructTokenBench test proteins are both PDB-derived and overlap isn't discussed (less critical for SSL, but worth stating).

Questions for authors
  1. Can you report block-wise results for a randomly-initialized encoder and for both AEs? If AE intermediate blocks match/exceed LeJEPA/JEPA blocks 4–6, the "bottleneck" is a property of the 8-D head, not of latent prediction — that would change the paper's conclusions substantially.
  2. Can you add a masked-reconstruction baseline with the same encoder, masking (e.g. 70% random), and number of presentations? Without it, how do you attribute gains to latent prediction rather than masking?
  3. Were the AE baselines trained with rotation augmentation? Are StructTokenBench inputs given in PDB frame to non-invariant encoders? This bears on the ±SO(3) comparisons and AE-vs-JEPA fairness generally.
  4. What is "Mutation" masking in Table 2? Does the EMA JEPA show the same index shortcut? Can you report index purity / length correlation for all 18 models, not just the illustrated pairs?
  5. Why does rotation augmentation hurt 5/6 flexibility scores for JEPA+SIGReg but help LeJEPA? Why does trained block 6 fall to 60.39 on Rep fold vs. 76.66 for random init (Table 4)?
  6. Is probe capacity/tuning identical across 8-D and 768-D inputs, and selected on validation data only?
3 comments

We thank the reviewer for their constructive feedback. We addressed a number of points in The Response to All Reviewers, covering framing (item 1), the Cα-conditioned AE headline (item 2), chance-level splits and fold/superfamily dependence (item 3), single seed and probe variance (items 4 and 5), training budgets, including LeJEPA at JEPA's budget (item 6) and PDB overlap (item 10). We answer the remaining points here.

  • Random 768-D block 6 beats the best final 8-D model. We compare each trained block with a randomly initialised encoder at the same width. Trained LeJEPA block 6 scores higher than random-init block 6 on 8 of 12 tasks and lower on none, so the block advantage is not explained by width alone. (Section 4.3, Figure 4, Appendix Table 4)
  • Online vs EMA weights. The autoencoders use EMA weights, as in La-Proteina, and the JEPA-based models use their trained encoder weights. Online and EMA weights tie on all 12 tasks for both the fully latent AE and LeJEPA. (Appendix E.7)
  • Cropped models trained only on chains of at least 96 residues. To separate cropping from the length filter, we trained cropped and uncropped LeJEPA on the same proteins of at least 96 residues, with the same local masking, batch size and steps, over three seeds. Under these controls, cropping removes the chain-length organisation but shows no downstream gain. (Response to All Reviewers, item 7; Section 4.2, Appendix Table 7)
  • Effective-rank jump confounded with batch size. The near-full effective rank of the cropped models comes with the larger batch size of 96, as Section 4.2 now states. In the matched control at batch size 16, cropped LeJEPA has an effective rank of 5.6, versus 7.5 for uncropped LeJEPA. (Response to All Reviewers, item 7; Section 4.2, Appendix C)
  • SIGReg over 16 eight-dimensional vectors is weak. Uncropped LeJEPA, trained with batch size 16, already reaches an effective rank of 7.26 of 8, so SIGReg over 16 vectors does not collapse the latent. Batch size 16 is the largest our resources allow for uncropped proteins. Cropping allowed us to apply SIGReg over 96 vectors, which raises effective rank further (7.6 to 7.9). (Section 4.2, Appendix C, Appendix Table 8)
  • No external reference point. As a diagnostic study of JEPA representations, the paper does not aim to rank models on the benchmark, so we use published results as reference points: SLAE and ESM3 and, following the reviewer's suggestion, AminoAseed, the tokenizer proposed with StructTokenBench. They are shown where task, split and metric match. (Appendix Figure 7)
  • Rotation hurts JEPA+SIGReg flexibility. We now report this result. With rotation, JEPA+SIGReg scores higher on BindInt and Homo, lower on FlexRMSF and ties on the other nine tasks. (Appendix E.4)

Q1. Can you report block-wise results for a randomly-initialized encoder and for both AEs? If AE intermediate blocks match/exceed LeJEPA/JEPA blocks 4–6, the "bottleneck" is a property of the 8-D head, not of latent prediction — that would change the paper's conclusions substantially.

We report block-wise results for a randomly initialised encoder and for all 12 blocks of both autoencoders, under the same protocol (Response to All Reviewers, item 6; Figure 4, Appendix Figure 9). The autoencoder blocks do not match the LeJEPA/JEPA blocks. The fully latent AE's blocks tie with the random encoder on 11 of 12 tasks, while the blocks of every JEPA-based model score higher than the fully latent AE's on at least five tasks and lower on none. The Cα-conditioned AE's blocks are the lowest on 10 of 12 tasks. Block 12 also exceeds the final latent on 10 to 11 tasks for the JEPA-based models but on only six to seven for the autoencoders, so the gap between the blocks and the final latent is larger for the JEPA-based models (Section 4.3).

Q2. Can you add a masked-reconstruction baseline with the same encoder, masking (e.g. 70% random), and number of presentations? Without it, how do you attribute gains to latent prediction rather than masking?

We thank the reviewer for this suggestion. We trained a masked AE with the same encoder, masking and training budget as JEPA-70 (Section 3.1). JEPA-70 and the masked AE tie on nine tasks. JEPA-70 scores higher on FlexRMSF, and the masked AE on FlexNEQ and Homo. LeJEPA scores higher than the masked AE on seven tasks and lower on none at 100k steps, and higher on four and lower on two at 25k steps (Section 4.3, Appendix Figure 9, Appendix Table 5).

Q3. Were the AE baselines trained with rotation augmentation? Are StructTokenBench inputs given in PDB frame to non-invariant encoders?

Yes to both. The autoencoders are trained with a random global rotation of each protein, and StructTokenBench gives all structures to every encoder in their PDB orientation. (Sections 3.1 and 3.4)

Q4. What is "Mutation" masking in Table 2? Does the EMA JEPA show the same index shortcut? Can you report index purity / length correlation for all 18 models, not just the illustrated pairs?

"Mutation" was an exploratory action-conditioned variant. It is outside the scope of this study, so we removed it from Appendix A and list mutation effects as future work. JEPA, with its EMA target encoder, shows no index shortcut (index purity 1.5% to 1.9%, random init 1.6%). The shortcut appears only in JEPA+SIGReg with absolute positions (94% to 96%). Index purity, chain-length correlation and effective rank for all 18 models are now reported in Appendix Table 8.

Q5. Why does rotation augmentation hurt 5/6 flexibility scores for JEPA+SIGReg but help LeJEPA? Why does trained block 6 fall to 60.39 on Rep fold vs. 76.66 for random init (Table 4)?

Interesting question. Once training-seed noise is taken into account, rotation lowers JEPA+SIGReg only on FlexRMSF and ties on FlexBFactor and FlexNEQ (Appendix E.4). For uncropped LeJEPA, rotation scores higher on eight to nine of the 12 tasks in each of three seeds and lower on at most one (Appendix Table 7). Rotation also raises the chain-length correlation of JEPA+SIGReg from 0.23 to 0.97, while uncropped LeJEPA is already at 0.97 without rotation (Appendix Table 8). Our hypothesis is that random rotation removes the orientation that context and target share, so the predictor relies more on protein size. (Appendix E.4).

On Rep, LeJEPA's block 6 is far below random init on the fold split but far above it on the superfamily split (Appendix Table 4), and averaged over both splits it ties with random init. We hypothesise that block 6 relies on chain-length cues that transfer within known superfamilies but not to unseen folds.

Q6. Is probe capacity/tuning identical across 8-D and 768-D inputs, and selected on validation data only?

Yes. The probe and its validation-based checkpoint selection are identical for 8D and 768D inputs, and both only use validation data (Section 3.4).

We thank the reviewer again and hope these revisions address the reviewer's concerns.

Thanks for the additional experiments—they address several of my concerns. Still, JEPA-70 does not clearly outperform the matched masked AE, and cropping shows no downstream benefit under controlled settings. The explanations for the rotation effects and the drop on the Rep fold split also remain tentative. Overall, I’m still not convinced that the results support the broader claims, so I’ll keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.