- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies representation similarity across PLM scales on proteins associated with IDR annotations. It proposes Pan-IDR, a contrastive protein–ligand retrieval model combining evolutionary pooling, functional attention, and dispersion regularization. The authors interpret increasing embedding similarity as representation collapse and a stronger ligand-only baseline as evidence of ligand shortcuts.
Critical strengths
- A relevant biological question. Understanding whether PLMs distinguish disordered regions and support ligand retrieval is worth investigating.
- Negative results are reported. The stronger simple baseline and weak contributions of several components could be informative if the dataset and evaluation are validated.
Critical weaknesses
- The data do not isolate IDR binding or representation. Targets are selected by matching DisProt proteins to PDBbind through UniProt IDs. A protein containing an IDR may bind its ligand through a folded domain, and the paper does not establish IDR-localized contacts. Embeddings also use full sequences truncated to 1,022 residues rather than annotated IDR segments. This undermines the central IDR-specific conclusions.
- The retrieval task and baseline inputs are unclear. A query is an unseen protein, yet a hit requires the retrieved ligand to match the query’s affinity class. Affinity belongs to a protein–ligand pair, so the query label and candidate ground truth need explicit definitions. It is also unclear how ECFP retrieves ligands from a protein-only query. Without comparable inputs and a valid metric, the ligand-shortcut conclusion is unsupported.
- Cosine similarity does not establish scaling-induced collapse. High raw cosine similarity can reflect anisotropy without loss of useful information. The paper lacks centered representations, effective-rank analysis, direct discrimination tests, and matched non-IDR controls. Moreover, similarity barely changes from 35M to 650M; three checkpoints do not establish a general scaling trend.
- The split description conflicts with the evaluation tiers. The paper claims cluster-disjoint splits at 30% identity with no homologous train–test targets, yet reports test tiers with 30–70% and over 90% similarity to training targets. Identity, alignment coverage, and clustering may differ, but these definitions must be reconciled before the zero-shot claims can be assessed.
- The methodological contribution is weak. Only EvoPool provides a substantial distant-tier benefit, and its relationship to the cited existing EvoPool method needs clearer differentiation. Functional attention and dispersion regularization add little. The full model’s weaker performance than ECFP does not itself demonstrate an intrinsic limitation of protein-aware learning.
- The presentation does not clearly connect the claims and experiments. The abstract combines scaling, collapse, model design, and dataset shortcuts without establishing a clear central argument. There is no overall architecture diagram explaining Pan-IDR and its evaluation. Table 3 reports uncertainty inconsistently across methods and metrics, while Figure 3 largely repeats its results.
Questions for authors
- What fraction of ligands contact annotated IDR residues, and do the findings hold when embeddings use only those regions?
- Can you provide one complete retrieval example, including the protein query, its label, candidate labels, hit calculation, and the inputs available to ECFP?
- How does the 30%-identity split coexist with the higher-similarity test tiers?
- Does the scaling result persist after correcting anisotropy and testing actual discrimination against matched non-IDR controls?
- Could you plz provide the raw code?