acceptodds
Withdrawn from ICLR 2027

ProtCompass: Biological Information and Predictive Utility in Protein Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Protein encoders are widely evaluated through structural probing, downstream prediction, and concept erasure, but these measurements can mix learned signal with information already present in the input or introduced by the evaluation itself. We introduce **ProtCompass**, a unified evaluation approach for separating these sources and identifying what protein representations genuinely contribute. ProtCompass audits what each encoder actually reads, compares structural probe scores with non-learned descriptions of the same inputs, evaluates downstream performance against simple sequence baselines, and controls concept-erasure experiments for the representation capacity removed together with the target property. Across diverse protein encoders and biological tasks, we find that structural probe scores can largely reflect retained input information, simple sequence statistics explain substantial downstream performance on several tasks, and many apparent task–property dependencies weaken once concept-erasure effects are properly controlled. These results provide a clearer way to determine what protein representations learn, what they add to biological prediction, and which properties downstream models actually use.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein representations, biological information, representation interpretability, sequence–structure representations, input determinacy, structural probing, concept erasure, feature accessibility, task-relevant information, matched controls, property repair, protein foundation models.

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_TNq3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProtCompass evaluates protein representations through input-matched structural probes, simple sequence baselines, and controlled concept erasure with property restoration. Across 33 encoders, 21 probes, and ten downstream tasks, it shows how input information and intervention effects can confound common interpretations of representation quality.

Critical strengths
  1. Careful controls for a relevant problem. Input audits, matched baselines, and erasure controls address real weaknesses in interpreting protein representations. The extensive follow-up analyses make this a substantial empirical study.
  2. Useful caution about information attribution. The reduction of apparent task–property dependencies under stronger controls is informative. The paper also acknowledges model heterogeneity and reports results that narrow its initial interpretations.
Critical weaknesses
  1. It is unclear how the framework helps users. The paper provides many diagnostics but no complete example showing how they lead to a better model choice or a new biological conclusion. The value of combining the three analyses is therefore unclear.
  2. The probing results have limited implications. An encoder scoring below a coordinate-based baseline does not mean its representation is less useful. The paper needs to explain what users should do with these scores.
  3. The model groups are not directly comparable. Sequence and structure encoders differ in size and pretraining. Among pretrained models with at least 100M parameters, the overall gap is not significant (p = 0.093). Rankings also change on residue-level tasks.
  4. The erasure results do not show what the original predictor uses. Removing equal variance may remove different amounts of useful information, and retraining the head changes the question being tested. The remaining stability–composition result is not significant in the fixed-head test (p = 0.22), while restoration mainly comes from parent-protein composition.
  5. The main takeaway gets lost in the details. The paper is well organized, but the abstract and main text need a few concrete findings that readers can use. Table 2 contains too much prose in small text and would work better as a short explanation or a simpler table.
Questions for authors
  1. Can you provide a worked example where ProtCompass changes a model-selection decision or biological interpretation relative to standard evaluation?
  2. What evidence shows that input-matched probing gives a more useful assessment of representation quality, beyond detecting information already supplied in coordinates?
  3. How should readers interpret property restoration when the fixed-head test is not significant and the repair mainly reflects parent-protein background?
Reviewer_81HuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate whether prevailing protein models learn representations that add predictive value beyond their input modalities (either sequence, structure, or both). To accomplish this, the authors establish modality-specific baselines that use simple summaries of the modality as predictors and use these baselines as "floors" that describe the information already trivially present within the inputs. Across a large set of protein models, the authors show that structure-based models rarely outperform structure-based floors for structure-based tasks, whereas sequence-based protein models do outperform sequence-based floors. The authors additionally perform an analysis of whether erasure of certain protein properties from model representations affects their performance on downstream tasks. Finally, the authors benchmark both randomly initialized and pretrained versions of many protein models on a range of downstream protein property prediction tasks,

Critical strengths
  1. The asked question is interesting: how much of protein model performance on downstream task can either be explained by simple input statistics? Establishing a meaningful "floor" for performance beyond the typical either inherent metric floor (e.g. 0.5 for AUROC) or a weak baseline floor like a randomized/shuffled input is valuable for interpreting model performance.
  2. Extremely exhaustive set of models evaluated (33 models) lends credence to findings
Critical weaknesses
  1. Key components of the study are completely undescribed in the main text. Lack of these details makes it hard to contextualize results of experiments. For example:
    • How input-matched floors/baselines are actually calculated. These appear to be described Appendix B.1 and on the 11th page of the Appendix in section B.9, but the presentation is unclear.
    • For the erasure analysis, what are the protein properties actually being erased? How were they selected?
  2. The work seems to mix two main goals:
    • Asking how much information protein models learn beyond simple statistics of their inputs (Section 3.1) and what that information is and how it's used (Section 3.3)
    • General benchmarking of protein models (Figures 3 and 5)

I think this lack of focus contributes to the manuscript generally being hard to parse

  1. Immense amounts of results are contained in the Appendix (~60 pages) which makes it quite difficult for the reader to understand what takeaway messages they should have from the work and how it should change their thinking or practice in the protein modeling space
Questions for authors

Questions affecting assessment

  1. Related to W1 above, can the authors provide a succinct description of what the input-matched floors are and how they're calculated as well as succinct description of what properties are being erased in the erasure analysis?
  2. Given the various results around performance of simple baselines, general benchmarking, and property erasure, how should a fellow researcher in protein models change their thinking around future development or usage of protein models?
  3. What is the purpose of the input audit step? It seems like this is an automated way to treat each model as a black box to see what aspects of the inputs it responds to, but I would have thought the inputs used would be clear from the model descriptions themselves. Are there cases where the input audit reveals a disconnect between the model descriptions and what inputs models actually respond to? If not, it seems to add an unnecessary amount of complexity.

Suggestions not affecting assessment:

  • The main results from Figure 2 are quite hard to parse: suggest moving the rows from 2b to be columns in 2a for easier comparison, possibly also add two columns representing mean over sequence models and mean over structure models, respectively
  • Methods are quite hard to follow, suggest adding a succinct high-level overview of the tests performed, e.g. as a Table or a bullet point list
  • Confusing/confounding to include randomly initialized models in the comparisons in Figure 3
    • Suggest changing naming scheme such that randomly initialized models have the special suffix (e.g. "-rand") instead of the pretrained ones, since the models that don't have a randomly initialized counterpart have no suffix, making it hard to tell which models are pretrained and which are not

Summary of review

The work as submitted is getting at interesting questions of understanding what information self-supervised protein models are learning beyond simple statistics of their input modalities. However, I think the current work attempts to ask and answer too many questions at a fairly shallow level whereas diving deeper into one of the constituent analyses could produce a work that would be both more focused, and thus easier to follow, and have concrete takeaways that would be meaningful to the community. I believe this is a clear reject in its current state.

Reviewer_CP3oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The combination of variance-matched null controls and property repair tests provides a well-motivated approach to evaluating task-specific information dependence.

  2. The paper provides extensive evaluations across diverse protein encoders and biological tasks. The additional control experiments and robustness analyses strengthen the empirical findings.

Critical strengths
  1. The combination of variance-matched null controls and property repair tests provides a well-motivated approach to evaluating task-specific information dependence. This improves upon conventional concept erasure by accounting for general representation loss.

  2. The paper provides extensive evaluations across diverse protein encoders and biological tasks. The additional control experiments and robustness analyses strengthen the empirical findings.

Critical weaknesses
  1. Although the paper provides some interesting observations, the biological insights are limited, and the practical value of the proposed framework is not sufficiently demonstrated.

  2. The manuscript is excessively lengthy and defensive, with numerous repetitive control experiments that obscure the central contributions. The anonymous code repository linked in the paper was inaccessible.

Questions for authors

See above.

Reviewer_pHZYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a new benchmark for evaluating protein encoders called ProtCompass. The objective is to eliminate common evaluation distortions like the learned signal in the protein representation with the information that is already present in the input. They provide an evaluation framework incorporating 33 encoders, 21 probes and 10 biological predictions. ProtCompass introduces matched non-learned reference baselines like sequence statistics to evaluate if the representations learn anything more than what can be easily inferred from the sequence. The paper shows several interesting results like sequence-only encoders achieve better performance on all 10 biological tasks.

Critical strengths

The overall idea of the paper is sound and the community will benefit from a benchmark like ProtCompass. The idea of Matched Non-Learned controls is novel and will help isolate information that is already included in the input. The authors perform empirical input audits through jittering coordinate swapping etc, which is also a good idea.

Critical weaknesses
  • It seems like the paper was written with signifcant AI help (especially in the appendix. Total page length is around 75 pages. The appendix shows tell-tale signs of written using AI (example: Table 2 "what each comparison in the paper establishes", "what the rule names")

  • The anonymous github repo link gives 404 error, but exposes the identity of the author(s)!

  • Given the extensive use of AI, it becomes hard to evaluate whether the authors have completely verified all the experiments. It seems like the authors have not taken the trouble to even write the experiments clearly or decide which experiments are relevant.

  • The paper makes very strong claims like: "EGNN declares that it reads coordinates and its embeddings are measured not to change when the coordinates change." It is hard to believe that dMaSIF or EGNN are unresponsive to any coordinate changes. These kind of strong claims require deeper evaluation.

Questions for authors
  • Is it possible to clearly establish that EGNN model is unresponsive to any coordinate changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.