- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates predictive latent representation learning for all-atom protein structures by adapting JEPA, JEPA+SIGReg, and LeJEPA to the La-Proteina encoder. Its main contributions are the identification of residue-index and chain-length shortcuts, protein-specific view designs intended to mitigate these shortcuts, and a layerwise analysis showing that intermediate representations often transfer substantially better than the final 8D latents. Validation uses frozen-encoder evaluation on StructTokenBench, supplemented by masking, positional encoding, cropping, rotation, and representation diagnostics. The results demonstrate useful downstream representations and informative failure modes, although performance is task-dependent and predictive objectives do not consistently outperform reconstruction baselines.
Critical strengths
Quantitative diagnosis of protein-specific shortcuts. The paper goes beyond downstream scores to identify residue-index organization in JEPA+SIGReg and chain-length organization in LeJEPA. In particular, replacing absolute positioning with relative positioning sharply reduces exact-index neighborhood purity while preserving effective rank and improving most downstream task/splits. This provides actionable evidence that apparently noncollapsed representations can nevertheless be dominated by shortcuts.
Thorough examination of where transferable information is retained. The layerwise analysis shows substantial downstream gains from intermediate representations, supported by comparisons against randomly initialized features and additional evaluation in JEPA. This is consequential because evaluating only the final latent would substantially underestimate the information learned by these encoders.
Critical weaknesses
The comparisons do not isolate the contribution of predictive pretraining. The strongest transfer results use intermediate 768D representations, but the reconstruction baselines are not subjected to the same layerwise analysis. Indeed, the fully latent autoencoder already achieves the highest average functional-site and flexibility scores among the reported 100k final-latent configurations. Without comparable intermediate-layer reconstruction baselines, it remains unclear whether the strongest results reflect a benefit specific to predictive objectives. Furthermore, the absence of sequence-only and backbone-only controls leaves the contribution of all-atom information unresolved. These omissions limit the methodological significance beyond applying existing objectives to a new input setting.
The chain-length mitigation results do not establish a clean link between shortcut removal and improved transfer. The controlled cropping comparison reduces chain-length organization but also sharply lowers effective rank. The configurations used for the main downstream results subsequently change local masking, batch size, training duration, and the eligible protein population. Consequently, their performance cannot be attributed specifically to removing the chain-length shortcut. Although these differences are disclosed, the missing controlled comparison and single-seed evaluation weaken a central practical conclusion about effective view design.
Questions for authors
Clarification: How are overlaps between the pretraining corpus and StructTokenBench test proteins assessed? Please distinguish exact-chain overlap from sequence-homology overlap. Also, are encoder layers and model configurations selected using downstream validation sets, or are the reported best-layer comparisons descriptive comparisons on test results?
Additional evidence that could change my assessment: How do the intermediate representations of the fully latent autoencoder compare with those of JEPA/LeJEPA under the same downstream protocol and comparable training budgets? This would help determine whether the strongest transfer results are specific to predictive pretraining.
Additional evidence that could change my assessment: For the final cropped configuration with 50% local masking, does chain-length organization remain low while effective rank remains high? Can its downstream performance be compared with an uncropped configuration matched for masking, batch size, eligible proteins, and training budget? Replication of this key comparison across seeds would strengthen the interpretation.
Additional evidence that could change my assessment: Are the main transfer gains retained when side-chain information is removed and only amino-acid identity and backbone coordinates are provided? This would clarify whether all-atom detail contributes materially to the reported representations.
2 comments
Thank you again for your review. As the discussion period closes in a couple of hours, we wanted to check whether our revision addressed your concerns. In short, the revision adds layerwise evaluation of all 12 blocks of the fully latent AE, the Cα-conditioned AE and a randomly initialised encoder (Figure 4, Appendix Table 5), the matched uncropped/cropped comparison over 3 seeds (Appendix Table 7), and backbone-only and sequence-only controls showing that side-chain atoms contribute significantly (Appendix Table 10). If there is anything else you would like us to clarify or test, please let us know and we will do our best to address it in the remaining time.
We are grateful to the reviewer for the clear and focused questions. We addressed a number of points in The Response to All Reviewers. Following the reviewer's suggestion, we now present the paper as a diagnostic study, as described in the Response to All Reviewers, item 1. We answer the remaining points here.
The overlap is addressed in the Response to All Reviewers, item 10, and Appendix E.3. As requested, we now report exact-chain overlap and sequence-homology overlap (MMseqs2) separately. Removing exact-chain overlaps changes the comparisons against the fully latent AE by at most one task.
Encoder layers are selected on the downstream validation sets. For each model and task, we report the test score of the block with the best validation score, and probe checkpoints are also chosen on validation data (Response to All Reviewers, item 6; Section 3.4). For 94 of 96 model-task pairs, the validation-chosen block is within training-seed noise of the best test block (Appendix Table 6).
As requested, we now evaluate all 12 blocks of the fully latent AE under the same downstream protocol as JEPA and LeJEPA, including LeJEPA at JEPA's training budget (Response to All Reviewers, item 6; Figure 4, Appendix Figure 9). The fully latent AE's blocks tie with a randomly initialised encoder on 11 of 12 tasks, while the blocks of every JEPA-based model score higher than the fully latent AE's on at least five tasks and lower on none. At the same training budget as JEPA, LeJEPA's blocks score higher than the fully latent AE's on six tasks and lower on none (Section 4.3, Appendix Table 5).
For the final cropped models trained with batch size 96, chain-length correlation stays low (0.19 to 0.51 across the four cropped models, versus 0.97 for uncropped LeJEPA) and effective rank stays near full (7.6 to 7.9 of 8; Appendix Table 8). As requested, we also ran the matched comparison over three seeds, with the same masking, batch size, eligible proteins and steps (Response to All Reviewers, item 7). Cropping lowers the chain-length correlation from 0.85 to 0.21 but does not improve downstream performance (Section 4.2, Appendix Table 7).
As requested, we trained JEPA+SIGReg on backbone atoms and amino-acid identity only, and on sequence only (Response to All Reviewers, item 9; Section 4.1). Side-chain atoms contribute substantially to the gains. Trained on backbone atoms and amino-acid identity, the final latent ties with random init on all 12 tasks. Trained on sequence only, it scores higher than random init on two tasks and lower on four (Appendix Table 10). Thus, all-atom detail contributes materially to the reported representations.
We appreciate the reviewer's suggestions and hope the revised manuscript answers the reviewer's questions.