- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ProtCompass evaluates protein representations through input-matched structural probes, simple sequence baselines, and controlled concept erasure with property restoration. Across 33 encoders, 21 probes, and ten downstream tasks, it shows how input information and intervention effects can confound common interpretations of representation quality.
Critical strengths
- Careful controls for a relevant problem. Input audits, matched baselines, and erasure controls address real weaknesses in interpreting protein representations. The extensive follow-up analyses make this a substantial empirical study.
- Useful caution about information attribution. The reduction of apparent task–property dependencies under stronger controls is informative. The paper also acknowledges model heterogeneity and reports results that narrow its initial interpretations.
Critical weaknesses
- It is unclear how the framework helps users. The paper provides many diagnostics but no complete example showing how they lead to a better model choice or a new biological conclusion. The value of combining the three analyses is therefore unclear.
- The probing results have limited implications. An encoder scoring below a coordinate-based baseline does not mean its representation is less useful. The paper needs to explain what users should do with these scores.
- The model groups are not directly comparable. Sequence and structure encoders differ in size and pretraining. Among pretrained models with at least 100M parameters, the overall gap is not significant (p = 0.093). Rankings also change on residue-level tasks.
- The erasure results do not show what the original predictor uses. Removing equal variance may remove different amounts of useful information, and retraining the head changes the question being tested. The remaining stability–composition result is not significant in the fixed-head test (p = 0.22), while restoration mainly comes from parent-protein composition.
- The main takeaway gets lost in the details. The paper is well organized, but the abstract and main text need a few concrete findings that readers can use. Table 2 contains too much prose in small text and would work better as a short explanation or a simpler table.
Questions for authors
- Can you provide a worked example where ProtCompass changes a model-selection decision or biological interpretation relative to standard evaluation?
- What evidence shows that input-matched probing gives a more useful assessment of representation quality, beyond detecting information already supplied in coordinates?
- How should readers interpret property restoration when the fixed-head test is not significant and the repair mainly reflects parent-protein background?