- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyzes the emergence of structure in protein and textual language models. Specifically, the authors look at the intrinsic manifold dimensionality (IMD, a measure of dimensionality of the embedding through layers) and local topilogical fidelity (LTF, a measure of embedding similarity in a given neighbourhood) as a function of the layer index. As with previous works in this field, the authors find that protein language models broadly expand and then compress the information in the embeddings. They also show that there are dinstinct 3D structure patterns in the 2 and 8 nearest neighbours, particularly for alpha helices. Finally, they show that IMD can be used in an unsupervised way to select the best layer for training supervised structural models.
Critical strengths
The authors expand the knowledge base for understanding emergence of structure in protein language models. They test the method on a large number of models and introduce two new variants of dimensionality measurement. They resolve a small open question from (Beshkov & Malthe-Sørenssen) about the peak in features near 8 nearest neighbours, being related to alpha helices. An unsupervised method to select the best embedding layer for downstream training is a potentially useful contribution to the field, and I appreciated the comparisons to different selection strategies.
Critical weaknesses
As acknowledged by the authors, the emergence of structure in LLMs is a well-studied phenomenon, and this paper provides limited additional insight. IMD is very similar to intrinsic dimensionality as explored in (Valeriani et al.) and LTF is very similar to the graph filtration moment (GFM) introduced in (Beshkov & Malthe-Sørenssen). Just because 3D coordinates get an average IMD of 3.88, doesn't mean that a value near 3.88 correponds to an internal representation of 3D structure. Additionally, the two LTF peaks near k=2 and k=8 can be trivially explained as the adjacent residues and the 4 nearest residues in a helix. The unsupervised selection method is interesting to me, but the tests in the main text mostly directly measure structural awareness, which the method claims to be directly selecting for anyway. This undercuts the "unsupervised" claim. I would like to see expanded testing of other labels such as EC and DeepLoc compared to strong baselines like the second last layer. Also, a small point, but the figures and labels/captions are very small and hard to read without massively zooming in.
Questions for authors
Have you checked if a simple model like sinusoidal Transformer positional encoding reproduces the LTF results? I am curious if this fully explains the k=8 peak.
Is there some more direct way to test if the IMD near 3.88 actually corresponds to improved structural understanding? Maybe reframing the layer selection for measuring contacts as an explicit test for this?
There seems to be a mismatch between Table 10 and Table 17 regarding worst regret for the final layer. I may be misunderstanding something, but can you explain why the average is greater than 1 while the individual values are all less than 1?
Do you plan to release public code to verify these results?
Does LTF fundamentally measure something different from GFM?
3 comments
R1 (Q2). Physical-IMD Proximity versus Native Structural Correspondence
| Component | Evaluation Protocol |
|---|---|
| Model coverage | All ten PLMs; every hidden layer, excluding the initial token-embedding layer |
| Structural calibration | Original 1,098-domain SCOPe reference for layer-wise IMD and mean LTF |
| Downstream evaluation | Original CATH test set of 75 protein domains |
| Contact-distance MAE | Frozen, layer-specific representations evaluated using the shared contact probe; results averaged over seeds 42, 2024, and 7 |
| Nonlocal Overlap@8 | Fraction of the eight nearest eligible native Cα neighbors recovered in representation space, excluding |
| Score aggregation | Distance MAE is pair-weighted within each seed; Overlap@8 is averaged within proteins and then equally across eligible proteins |
| Statistical analysis | Spearman correlation across hidden layers within each model; model-wise medians summarize the ten-model panel |
Importantly, the correlations are computed separately within each PLM, rather than pooling layers from architectures with different depths and representation scales. The model-wise median then gives equal weight to each backbone, regardless of its number of hidden layers.
IMD and LTF are evaluated using the SCOPe structural reference, while contact-distance prediction and nonlocal neighbor retrieval are evaluated on CATH. The latter provides a direct, probe-free geometric diagnostic alongside the supervised structural prediction results.
3.2. Physical-IMD Proximity Shows Weak and Inconsistent Structural Associations
Table rmYA-R1-B. Full-depth Spearman correlations between geometric indicators and structural performance. D denotes |IMD − 3.88|. Lower MAE and higher nonlocal Overlap@8 indicate better structural performance. Each correlation is computed across the hidden layers of the corresponding model.
| Model | Layers | ρ(D, MAE) | ρ(D, Nonlocal Overlap) | ρ(LTF, MAE) | ρ(LTF, Nonlocal Overlap) |
|---|---|---|---|---|---|
| ESM2-35M | 12 | +0.18 | −0.14 | −0.52 | +0.61 |
| ESM2-150M | 30 | +0.09 | −0.08 | −0.68 | +0.75 |
| ESM2-650M | 33 | +0.12 | −0.06 | −0.74 | +0.81 |
| Ankh Base | 48 | −0.07 | +0.03 | −0.59 | +0.70 |
| Ankh Large | 48 | +0.04 | −0.02 | −0.65 | +0.78 |
| ProtGPT2 | 36 | +0.29 | −0.22 | −0.47 | +0.57 |
| CARP-38M | 16 | −0.16 | +0.12 | −0.43 | +0.52 |
| CARP-76M | 33 | +0.06 | −0.10 | −0.62 | +0.72 |
| CARP-640M | 56 | +0.21 | −0.17 | −0.71 | +0.80 |
| ProtMamba | 16 | −0.09 | +0.07 | −0.56 | +0.65 |
| Model-wise Median | — | +0.075 | −0.070 | −0.605 | +0.710 |
| Physical-IMD proximity exhibits weak and inconsistent associations with structural performance across the ten PLMs. Its median correlation with contact-distance MAE is only +0.075, while its median correlation with nonlocal Overlap@8 is −0.070. The associations also change sign across architectures, indicating that closer agreement with the physical IMD reference does not consistently identify layers with better native structural correspondence or downstream distance prediction. |
Some models nevertheless show modest associations in the expected directions, including ProtGPT2 (ρ = +0.29 with MAE) and CARP-640M (ρ = +0.21). Thus, the results do not suggest that global dimensionality is entirely unrelated to structural utility. Rather, proximity to the particular physical reference value of 3.88 is not a reliable monotonic predictor of structural performance across depth.
In contrast, LTF exhibits substantially stronger and directionally consistent associations across all ten models. Its correlations with contact-distance MAE range from −0.74 to −0.43 (median −0.605), while correlations with nonlocal Overlap@8 range from +0.52 to +0.81 (median +0.710).
These results show that layers with stronger native-neighborhood fidelity generally support more accurate distance prediction and recover more sequence-distant native neighbors. The association with supervised contact-distance MAE is particularly informative because it evaluates the utility of frozen representations through a downstream predictive task, rather than solely comparing alternative neighborhood statistics.
R1 (Q2). Physical-IMD Proximity versus Native Structural Correspondence
5. Scientific Interpretation and Relation to Prior Work
Together, our results distinguish global shape-spectrum organization from the preservation and downstream accessibility of native structural relationships.
IMD measures the effective dimensionality of variation across protein representation shapes. Its layer-wise evolution provides a useful description of geometric expansion and compression, but proximity to the physical reference of 3.88 does not establish whether particular native residue relationships are preserved.
This distinction is supported by two direct observations. First, physical-IMD proximity exhibits weak and inconsistent associations with structural performance across all ten PLMs, whereas LTF shows substantially stronger and directionally consistent associations with both contact-distance MAE and nonlocal native-neighbor retrieval (Table rmYA-R1-B). Second, layers with similar IMD proximity can retain different levels of structural fidelity and predictive utility, as demonstrated by the matched-proximity comparisons (Table rmYA-R1-C).
These findings are consistent with prior studies examining the relationship between representation dimensionality and model behavior. Ansuini et al. [1] investigate intrinsic dimensionality in deep neural networks, while Valeriani et al. [2] analyze dimensionality patterns in Transformer representations. Their work motivates the study of geometric reorganization, but does not imply that matching a particular physical shape-spectrum statistic guarantees residue-level structural correspondence. Moreover, our SRV tangent-covariance participation ratio differs from the TwoNN-based intrinsic-dimensionality estimator used by Valeriani et al.
Beshkov and Malthe-Sørenssen [3] further connect protein representation geometry with SRV-based shape analysis and neighborhood-graph measurements. Our additional experiments clarify the empirical relationship between these levels of analysis: global dimensionality describes representation-shape variation, whereas native-neighborhood fidelity provides a more direct indicator of residue-level structural correspondence and its association with downstream structural prediction.
We do not claim that IMD is unrelated to structural information, that LTF causally determines predictive performance, or that LTF is fundamentally distinct from closely related neighborhood-based measures such as GFM. Rather, the present experiments establish why matching global dimensionality is insufficient as a standalone test of native structural understanding, and why direct structural correspondence must be evaluated separately.
Overall Response to Q2
Our results directly address the reviewer's proposed test: an IMD value near the physical reference of 3.88 does not, by itself, establish improved native structural understanding.
The existing ten-model contact evaluation shows that LTF-based selection achieves lower mean distance MAE than physical-IMD-based selection (8.52 versus 9.04 Å). Our additional full-depth analysis further demonstrates that physical-IMD proximity has weak and inconsistent associations with structural performance (median Spearman ρ = +0.075 for MAE and −0.070 for nonlocal overlap), whereas LTF shows substantially stronger and directionally consistent associations (−0.605 and +0.710, respectively).
The matched-proximity analysis provides complementary evidence: even among layers with closely matched distances to the physical IMD reference, native-neighborhood fidelity can distinguish structural correspondence and, in three of four evaluated models, is associated with better contact-distance prediction.
The resulting scientific distinction is clear: IMD characterizes global geometric organization, LTF measures native residue-neighborhood correspondence, and supervised contact prediction evaluates the accessibility of structural information for a downstream task. These properties are related but not interchangeable.
We will incorporate the full-depth and matched-proximity analyses into the revised manuscript and clarify that physical-IMD proximity is a descriptive comparison criterion, not a sufficient indicator of native structural understanding.
References
[1] Ansuini, A., et al. (2019). Intrinsic Dimension of Data Representations in Deep Neural Networks. Advances in Neural Information Processing Systems, 32.
[2] Valeriani, L., et al. (2023). The Geometry of Hidden Representations of Large Transformer Models. Advances in Neural Information Processing Systems, 36.
[3] Beshkov, K., and Malthe-Sørenssen, A. (2026). Towards Understanding the Shape of Representations in Protein Language Models. International Conference on Learning Representations.
R1 (Q2). Physical-IMD Proximity versus Native Structural Correspondence
LTF-based selection achieves lower contact-distance MAE than IMD-based selection across all ten PLMs, reducing the panel mean from 9.04 to 8.52 Å. IMD selection also improves modestly over the final layer in nine of ten models, suggesting some practical utility as a layer-selection heuristic, but not establishing that proximity to the physical IMD reference directly measures native structural correspondence.
A particularly informative case is ESM2-650M, where the different structural metrics do not favor the same representation.
Table rmYA-R1-A2. ESM2-650M structural evaluation at selected layers. Mean LTF is computed on SCOPe over k = 1–19. CATH distance MAE evaluates residue pairs with |i−j| > 6, while long-range contact P@L uses |i−j| > 24.
The IMD-selected layer (L23) yields substantially higher distance error than both the LTF-selected layer (L25) and the final layer (L33), despite being closest to the physical IMD reference. Its MAE of 8.75 Å exceeds the corresponding values of 7.29 Å and 7.97 Å, respectively.
At the same time, IMD selection achieves a slightly higher long-range P@L (0.52) than LTF selection (0.51). This small descriptive difference illustrates that representation utility depends on the particular structural endpoint; it does not establish a statistically reliable advantage for either selector in long-range contact precision.
Taken together, these results demonstrate that matching the physical IMD reference does not guarantee superior native structural prediction. IMD proximity can identify useful intermediate representations, but it does not consistently identify the layers that minimize distance-prediction error. Conversely, LTF-based selection performs better on contact-distance MAE across the evaluated panel without implying superiority on every structural metric.
Importantly, the selected-layer comparison tests the practical utility of the two selection criteria, but does not establish how IMD proximity relates to structural performance throughout the complete representation hierarchy.
We therefore extend this analysis across every hidden layer of all ten PLMs, directly testing whether proximity to IMD ≈ 3.88 tracks contact-distance MAE and sequence-nonlocal native-neighbor retrieval, and comparing these associations with those obtained from LTF.
3. Does Physical-IMD Proximity Predict Structural Performance Across Depth?
To directly test whether proximity to the physical IMD reference predicts structural utility, we extend the selected-layer comparison to every hidden layer across all ten PLMs. This analysis evaluates the relationship between global spectral dimensionality, native-neighborhood correspondence, and downstream structural prediction without restricting the comparison to layers chosen by either selector.
For model m at layer ℓ, we define physical-IMD proximity as:
D(m, ℓ) = |IMD(m, ℓ) − 3.88|A smaller D indicates closer agreement with the physical IMD reference. If this proximity reliably reflects structural utility, D should correlate positively with contact-distance MAE (lower is better) and negatively with nonlocal native-neighbor overlap (higher is better). Conversely, higher LTF is expected to associate with lower MAE and greater native-neighbor overlap.
3.1. Full-Depth Evaluation Protocol
We compute layer-wise associations using the original structural calibration set and the same downstream evaluation protocol as the manuscript.
Table rmYA-R1-0. Protocol for the full-depth structural correspondence analysis.