acceptodds
Withdrawn from ICLR 2027

Scaling Worsens Representation Collapse in Intrinsically Disordered Regions: Pan‑IDR, Evolution‑Guided Contrastive Learning and Ligand‑Dominant Shortcuts for Zero‑Shot Pan‑Cancer IDR‑Ligand Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning discriminative representations of intrinsically disordered regions (IDRs) remains a fundamental challenge for protein representation learning. Although IDR targets are commonly assumed to benefit from protein-aware representations, their low-complexity sequences may fundamentally limit the discriminative information available to protein language models. Here, we identify systematic representation collapse across amino-acid composition, ProtBERT, and ESM-2 embeddings on 543 pan-cancer IDR targets. Surprisingly, scaling does not alleviate this problem: increasing ESM-2 from 35M to 3B parameters raises mean pairwise cosine similarity from 0.870 to 0.944, indicating that larger models capture shared IDR sequence statistics more precisely without learning more discriminative representations. We introduce Pan-IDR, a contrastive learning framework combining evolution-guided aggregation (EvoPool), functional attention, and dispersion regularization. Under 5-fold zero-shot evaluation, Pan-IDR achieves a distant-target hit@1 of 0.339, compared with a 0.200 random baseline. Ablations further show that EvoPool is the only component providing a substantial improvement, suggesting that downstream objectives alone cannot recover discriminative information from collapsed representations. More strikingly, a ligand-only ECFP baseline using no protein information outperforms Pan-IDR, achieving distant-target hit@1 of 0.417 versus 0.339. This exposes a ligand-dominant shortcut in which ligand structural similarity provides a stronger predictive signal than learned protein–ligand compatibility, indicating that current IDR affinity data do not support robust protein-aware zero-shot generalization. Together, our results reveal a scaling-resistant limitation of current protein representation learning for IDRs and motivate representations and benchmarks specifically designed for disordered proteins.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation collapse, intrinsically disordered regions, protein language models, contrastive learning, zero-shot retrieval, metric learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PLbSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies representation similarity across PLM scales on proteins associated with IDR annotations. It proposes Pan-IDR, a contrastive protein–ligand retrieval model combining evolutionary pooling, functional attention, and dispersion regularization. The authors interpret increasing embedding similarity as representation collapse and a stronger ligand-only baseline as evidence of ligand shortcuts.

Critical strengths
  1. A relevant biological question. Understanding whether PLMs distinguish disordered regions and support ligand retrieval is worth investigating.
  2. Negative results are reported. The stronger simple baseline and weak contributions of several components could be informative if the dataset and evaluation are validated.
Critical weaknesses
  1. The data do not isolate IDR binding or representation. Targets are selected by matching DisProt proteins to PDBbind through UniProt IDs. A protein containing an IDR may bind its ligand through a folded domain, and the paper does not establish IDR-localized contacts. Embeddings also use full sequences truncated to 1,022 residues rather than annotated IDR segments. This undermines the central IDR-specific conclusions.
  2. The retrieval task and baseline inputs are unclear. A query is an unseen protein, yet a hit requires the retrieved ligand to match the query’s affinity class. Affinity belongs to a protein–ligand pair, so the query label and candidate ground truth need explicit definitions. It is also unclear how ECFP retrieves ligands from a protein-only query. Without comparable inputs and a valid metric, the ligand-shortcut conclusion is unsupported.
  3. Cosine similarity does not establish scaling-induced collapse. High raw cosine similarity can reflect anisotropy without loss of useful information. The paper lacks centered representations, effective-rank analysis, direct discrimination tests, and matched non-IDR controls. Moreover, similarity barely changes from 35M to 650M; three checkpoints do not establish a general scaling trend.
  4. The split description conflicts with the evaluation tiers. The paper claims cluster-disjoint splits at 30% identity with no homologous train–test targets, yet reports test tiers with 30–70% and over 90% similarity to training targets. Identity, alignment coverage, and clustering may differ, but these definitions must be reconciled before the zero-shot claims can be assessed.
  5. The methodological contribution is weak. Only EvoPool provides a substantial distant-tier benefit, and its relationship to the cited existing EvoPool method needs clearer differentiation. Functional attention and dispersion regularization add little. The full model’s weaker performance than ECFP does not itself demonstrate an intrinsic limitation of protein-aware learning.
  6. The presentation does not clearly connect the claims and experiments. The abstract combines scaling, collapse, model design, and dataset shortcuts without establishing a clear central argument. There is no overall architecture diagram explaining Pan-IDR and its evaluation. Table 3 reports uncertainty inconsistently across methods and metrics, while Figure 3 largely repeats its results.
Questions for authors
  1. What fraction of ligands contact annotated IDR residues, and do the findings hold when embeddings use only those regions?
  2. Can you provide one complete retrieval example, including the protein query, its label, candidate labels, hit calculation, and the inputs available to ECFP?
  3. How does the 30%-identity split coexist with the higher-similarity test tiers?
  4. Does the scaling result persist after correcting anisotropy and testing actual discrimination against matched non-IDR controls?
  5. Could you plz provide the raw code?
Reviewer_Xw6nOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to compare different featurization strategies for predicting protein-ligand binding, where the protein target is intrinsically disorder.

Critical strengths

None

Critical weaknesses

The paper appears heavily AI generated and AI written. The topic appears unsuited for ICLR, since (as far as I can tell) the experiments simply compare different featurization strategies.

Questions for authors

I would encourage the authors to put more effort into their submissions - AI-generated content is of little value.

Reviewer_6njWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors present a study on how well various protein sequence representations are able to discriminate between distinct intrinsically disordered regions (IDRs) across proteins. The focus on IDRs is motivated by their role in the molecular function of many disease-relevant proteins, with some proteins being largely disordered. The authors show that the tested sequence representation methods produce highly similar representations (as measured by cosine similarity) across distinct IDRs and thus are generally unable to discriminate between different IDRs. They also show that this representational collapse is also not alleviated by increased model scale. The authors then develop the Pan-IDR model, which combines a small molecule ligand encoder with a protein language model via a contrastive objective to produce a model that can retrieve ligands given an IDR target. Benchmarking across 4,762 IDR-ligand complexes (consisting of 543 unique IDR targets) shows that their method outperforms a random baseline but lags behind a ligand-only approach, suggesting that, in this setting, ligand structure largely determines IDR-ligand binding affinity.

Critical strengths
  1. The work is well-motivated: a large focus of the field is combining learned representations of proteins and small molecules to identify ligands with high binding affinity to a protein target.
    • Many disease-relevant protein targets are highly disordered or contain highly disordered regions, which both lack stable structures and tend to be composed of low-complexity sequence, making it difficult to learn meaningful representations using sequence- or structure-based self-supervised models
    • Approaches attempting to prioritize protein-ligand pairs implicitly assume protein models can distinguish between different IDRs
  2. Finding of ligand-dominant shortcut is interesting and has important implications for thinking through how to construct IDR-binding benchmarks in the future
Critical weaknesses
  1. Extensive formatting and presentation issues call into question whether or not authors proofread the manuscript before submission
    • Consistent misuse of \cite instead of \citep throughout hampers readability of the text
    • Repetition of results across tables and figures: each pair of Table/Figure 2, 3, and 4 show the same results
    • Repetition of text throughout, e.g. first paragraph of section 4.1 has the same sentence repeated back-to-back, explanation of poor scaling of ESM-2 models repeated in 3rd paragraph of intro, Sections 2.1, 4.2
    • Clipping of figure elements in Figure 3
  2. A core claim of the paper is that current protein models poorly capture the diversity of IDRs, however I think the validity of this claim is hard to assess in absence of additional context (see questions below)
  3. The other core component of the paper, the development of the Pan-IDR approach for ligand-conditioned IDR target retrieval, is poorly supported due to the sole comparisons being a random baseline and a ligand-only baseline
Questions for authors

Questions affecting assessment

I've tried to order the questions below roughly in order of priority and by how much work I believe it would take to answer them:

  1. How exactly is "pan-cancer" defined? Or put another way, what makes these IDR targets "pan-cancer targets"?
  2. The distinction between intrinsically disordered regions (IDRs) and intrinsically disordered proteins (IDPs) is not really clear: when constructing IDR representations, are residue-level embeddings being aggregated over the entire protein or just the IDR?
  3. While the representational collapse metrics seem quite high (average cosine sims > 0.8, large fraction of pairs with sim > 0.9), some baseline comparisons would help contextualize these numbers.
    1. How should we think about the dynamic range of these pairwise similarities? A "positive control" could be pairwise similarity of homologs defined at the structure level (e.g. via FoldSeek) with a range of sequence identities. A "negative control" could be proteins that are distinct at both the structure and sequence level
    2. How surprising is representational collapse here? If IDR sequences are treated in isolation and are highly similar the sequence level, then PLMs like ESM2 and ProtBert will by definition produce highly similar representations. This will also be true if IDRs are contextualized within the full protein if the proteins themselves are share high sequence similarity. Metrics summarizing distributions of alignment-based metrics like % identity and % overlap between the 543 targets would be illuminating.
  4. Related to W2: do the authors see the same representational collapse with more recent methods that purport to alleviate poor scaling in PLMs, such as ESM-C or reverse-distilled ESM-2?

Questions not affecting assessment:

  • In section 4.5, the authors suggest that strong binders tending to share structural motifs may drive the ligand-dominant shortcut, can this be quantified? e.g. by clustering ligands by Tanimoto similarity and annotating max affinity across all targets

Summary of review

I think the work is well-motivated, especially as the field moves from largely modeling proteins as static structures to thinking carefully about protein dynamics and disordered regions. The authors' question of how well current approaches can actually represent distinct IDRs is definitely an interesting one. However, I think the current work fails to make a strong case for its finding: metrics of representational collapse lack context and Pan-IDR performance is only compared to a random baseline. The paper is further hampered by severe presentation issues. For these reasons, I recommend a rejection in its current state. I suggest that the authors focus on fleshing out the narrative around IDR representation collapse and what this means for current state-of-the-art protein language models to develop the manuscript into one that will surely be of interest to the community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.