acceptodds
Under review as a conference paper at ICLR 2027

What Goes Up Should Come Down: A Training-Distribution Control for Interpretability Metrics

Abstract

Interpretability metrics are often supported by showing that their results are reproducible across seeds, model conditions, and analytical choices. Such robustness does not establish that a metric is valid for the interpretation attached to it. We demonstrate this distinction using a relation-referenced metric for sparse autoencoder (SAE) features in protein language models. The metric, , measures whether feature activations cluster across residues in spatial contact. In a controlled comparison of masked and causal models, gives a plausible masked-model advantage at every sampled depth. The difference reproduces across model seeds, persists under prediction matching, and survives six diagnostic analyses spanning dictionary fitting, reconstruction quality, residue selectivity, and representation basis. However, after retraining the same models on sequences with native residue order destroyed, increases in 52 of 54 objective-by-depth-by-seed cells. Random-initialisation and no-model baselines reveal complementary problems: the trained masked-model score does not exceed the random-initialisation range, and a cysteine indicator exceeds every learned feature in 32 of the same 54 cells. These results do not rule out structural information in , but they show that its robust masked–causal difference is not specific evidence of structural organisation learned from native residue order. Applying the same framework to further published interpretability metrics reveals different patterns of failure across the training-distribution, random-initialisation, and no-model controls. Interpretability metrics therefore require controls matched to each link between a score and its intended interpretation; reproducibility within a measurement pipeline is not sufficient evidence of construct validity.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, sparse-autoencoders, evaluation, protein language models, controls, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_17LQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The presented paper tests whether reproducible interpretability scores support their intended biological interpretation or not. In paired masked and causal pLMs the structural co-activation metric L_struct consistently favors masked models. Yet training on residue-permuted sequences increases the score in 52/54 comparisons. Randomly initialized models and amino-acid indicators provide further counter-examples.

Critical strengths
  • The paired training design controls architecture, initialization, corpus, and batch order. Replicated re-training interventions provide substantially stronger evidence than another comparison between unrelated pre-trained models.

  • The combination of shuffled training, random initialization, and deterministic indicators clearly demonstrates that a high structural co-activation score need not establish learning from native residue order. Sensitivity analyses address several straightforward implementation explanations.

Critical weaknesses
  • The conceptual novelty is limited. Shuffled pre-training already exposed problems with syntactic probes; random-model failures of SAE metrics were already demonstrated, where scores similar to those from trained models were reported. The main advance here is the controlled protein-specific counter-example and the combination of controls.

  • The central counter-example is stronger than the explanation. Shuffling preserves composition, changes the evaluation distribution relative to training, and improves SAE reconstruction in every matched cell; reconstruction improvement correlates with the score increase. The limited dictionary-free checks help, but the paper does not establish which preserved or altered property drives the result. This supports the narrow conclusion about score specificity, while leaving the underlying failure unresolved.

  • Broader metric failure claim is insufficiently justified. Composition and taxonomy information can survive order destruction, so annotation F1 or decoder-geometry agreement need not decrease unless their intended interpretation specifically requires native order. Several implementations and label sets differ from the original evaluations. Moreover, no statistically detectable decrease with three seeds does not establish insensitivity. These tests support metric-dependent responses more clearly than general invalidity.

Questions for authors
  • Do independent structural readouts decrease after shuffled training, particularly separation-matched contact probes? Report both raw representations and SAE latents.

  • Does the score increase persist after matching SAE reconstruction quality? Can a residue-conditioned permutation null separate biochemical clustering from contextual grouping?

  • For each additional metric, what precise published interpretation requires its score to decrease under this intervention? Please distinguish significant increases, inconclusive changes, and baseline failures, using common concept sets.

  • Why does the exact sign test treat the 6 objective-by-seed summaries as independent when paired objectives share initialization and training inputs? Report inference accounting for these three paired seed clusters.

6 comments
AuthorsOpenReview ↗

First Reply To Reviewer 17LQ

Thank you for a precise review. We ran the analyses you asked for and report them below. We label your Critical Weaknesses W1–W3 and Questions Q1–Q4, and answer in five parts, pairing overlapping points: W1 here; W2 with Q2 in part 2; Q1 in part 3; W3 with Q3 in part 4; Q4 in part 5.

W1 (novelty). Shuffled pre-training (Sinha et al., 2021), random-model baselines (Heap et al., 2025) and label randomisation (Adebayo et al., 2018, which we will cite) are precedents. We differ in two ways: the intervention removes one targeted relation (native residue order) while keeping the properties that could otherwise produce the score (length and composition), and the criterion is directional (a measure of a relation learned from data should not rise when that relation is removed).

  1. It catches what the usual checks do not. LstructL_{\text{struct}} reproduces across seeds and dictionaries, survives prediction matching and six diagnostics, and still rises after order destruction in 52 of 54 cells, and in 3 of 3 new ESM-2-8M-architecture pairs trained about 240 times longer.
  2. It is not replaced by a random-model baseline. At 32× the training budget the masked model scores above every random-initialisation draw at blocks 11 and 14, yet order destruction still raises the score in all six cells (Tables 8 and 11).
  3. It changes the conclusion. Without it, the masked–causal difference would read as better structural organisation learned from native order.
  4. It shows what each score responds to. Combined with the random-initialisation and no-model baselines, a residue-conditioned null, and scoring against an unrelated protein’s structure, it separates scores that track learned information from scores that track residue identity or position along the sequence. The annotation F1 metrics fall when order is removed from models that have learned it. At the primary budget, LstructL_{\text{struct}} rises through residue identity and position acting on real contacts, and dstructd_{\text{struct}} in causal models through features that vary smoothly along the chain (parts 2 and 4).
AuthorsOpenReview ↗

Second Reply to Reviewer 17LQ

W2 and Q2 (mechanism, matched reconstruction, and residue-conditioned null). Thank you for raising both points. The new analyses below show that the increase survives matched reconstruction, and that a residue-conditioned permutation null attributes most of it to residue identity and position.

  • Reconstruction does not explain the increase (Q2, first part). The order-destroyed model's dictionary reconstructs better. We have now run the matching proposed in the paper's Discussion.
    • We stopped each order-destroyed dictionary at the first step where its held-out explained variance reached that of the native model's dictionary. LstructL_{\text{struct}} still rises (blocks 11, 14 and 18): masked 3/3 seeds, +0.0110, p = 0.049; causal 3/3, +0.0275, p < 0.001.
    • It also rises against native dictionaries stopped at the same training step (masked 3/3, p = 0.035; causal 3/3, p < 0.001), and on the raw residual stream with no dictionary (causal +0.0217, p = 0.002; masked +0.0041, p = 0.11; 3/3 seeds positive in each).
    • Sparsity is another free analysis choice that shifts metric verdicts, reinforcing §3.5. The paper fixed k=256k = 256 active latents. The causal rise reproduces at k=64k = 64 and 128128 and without a dictionary (p ≤ 0.038; masked same direction, n.s.). At k=32k = 32 the causal direction reverses (0/3 seeds, p = 0.009; repeat p = 0.055) while the masked rise keeps its direction (3/3, p = 0.15). Because four of the five published metrics, computed with their authors' functions where code exists, also change verdict on at least one check across kk (all but single-latent F1; W3 and Q3), controls must be run at the exact dictionary settings used in an evaluation; we will add the full kk-grid to the revision.
  • Composition: a stratified permutation null removes the masked increase and most of the causal one (Q2, second part). LstructL_{\text{struct}} compares each feature with a null in which its activations are shuffled among all residues of a protein, keeping the contacts fixed. The stratified null shuffles them only among residues of the same amino acid (the amino-acid null), or of the same amino acid in the same fifth of the protein's length (the amino acid × position null).
    • Masked: the amino-acid null removes the increase. With our dictionaries it goes from +0.0139 to +0.0009 (p = 0.80), and 1 of 3 seeds stays positive. On the raw residual stream, with no dictionary, it goes from +0.0041 to −0.0005 (p = 0.16), and 0 of 3 seeds stay positive.
    • Causal: the amino-acid null leaves most of the increase (raw residual stream: +0.0217 to +0.0179, 3 of 3 seeds still positive, p = 0.004; our dictionaries: +0.0288 to +0.0218, p < 0.001). The amino acid × position null removes the rest with our dictionaries (+0.0288 to −0.0003, 0 of 3 seeds positive, p = 0.18) and reduces it to +0.0013 on the raw residual stream (p = 0.13); results on further held-out partitions will be in the revision.
    • Planted contact features (289 synthetic features; in every protein each is on at one residue and at its contacts, off elsewhere; construction in the revised appendix) keep a median of 66% of their score under the amino acid × position null (10th–90th percentile 64–68%), while the order-destroyed increase largely disappears. So the null removes about a third of a known contact signal but most of the increase.
    • Amino-acid identity alone predicts more of the activations of order-destroyed models than of native ones (share of held-out variance, raw residual stream, native to order-destroyed: masked 0.74 to 0.97; causal 0.16 to 0.45; 3 of 3 seeds, p ≤ 0.004).
  • Real contacts: the increase depends on them (W2). Scored against an unrelated protein's structure (same length, another SCOPe fold; 136 of the 151 held-out domains have one), the causal increase falls from +0.026 to +0.004 (p = 0.020) and the masked one from +0.016 to +0.005 (p = 0.072). Because the null ignores sequence separation, it penalises features that vary smoothly along the chain. Referencing each score to its value against unrelated structures removes this bias, and the increase stays positive (masked +0.011, p = 0.021; causal +0.022, p = 0.004; on 400 dictionary-fitting domains +0.009 and +0.012, p ≤ 0.043). With the stratified null, this points to residue identity and position acting through real contacts.
  • Shift in evaluation distribution: it does not explain the increase on its own. The order-destroyed models are scored on native proteins they never saw, as in any training-distribution control (Sinha et al.'s shuffle-trained models also show much higher perplexity on natural text). But the mismatch does not raise every readout: on the same inputs, structure probes of the order-destroyed models fall to the random-initialisation level (Q1). And with less mismatch, the block-shuffled models of §2.2 (16-residue segments reordered, order kept within each) still raise LstructL_{\text{struct}} in all 9 causal cells and 6 of 9 masked cells (§3.2).
AuthorsOpenReview ↗

Third Reply To Reviewer 17LQ

Q1 (structural readouts, raw and SAE). We ran these readouts on the order-destroyed models. On both the raw residual stream and SAE latents, the structural readouts of §3.6 fall back to the random-initialisation level after order destruction. Below, "layer" means the paper's blocks 11, 14 and 18.

  • Residue-level probes (helix, strand, burial; the protocol of Table 14, unchanged).
    • Causal: order-destroyed models lose 86–97% of the native models' gain over random initialisation (layer means; 65–113% across the 9 seed-by-layer cells per label).
    • Masked: order-destroyed models lose 95–106% at layer 18. At layers 11 and 14 the native masked models are themselves barely above random initialisation on these probes (Appendix I), so there is little to lose.
  • Pairwise contact probe (§3.6, Appendix I): a logistic-regression probe on pairs of residue representations, in which each contact is matched with a non-contact from the same protein at the same sequence separation. We say the probe falls after order destruction when the native model scores higher than the order-destroyed one: the bootstrap interval for native minus order-destroyed lies above 0 in at least 2 of 3 seeds (rule fixed before the runs).
    • Raw residual stream: order-destroyed models are at the random-initialisation level in all 6 objective-by-layer cells. By the rule, the probe falls at masked layer 18 and causal layers 11 and 18.
    • SAE latents (our dictionary recipe): the same picture. Order-destroyed models are at the random-initialisation floor in all 6 of 6 cells, and by the rule the probe falls in 4 of 6 cells: masked layer 18 and causal layers 11, 14 and 18 (native AUROC 0.531–0.565 against random initialisation 0.527–0.538).
  • Contrast with LstructL_{\text{struct}}. On the exact representations where LstructL_{\text{struct}} rises after order destruction, supervised structural probes drop to the random-initialisation floor. (Note that at this training budget, native contact probes themselves only reach the ~0.59 AUROC of a model-free amino-acid-pair and sequence-distance baseline on a different split—yet unlike LstructL_{\text{struct}}, they never rise significantly when sequence order is removed.)
Reviewer_tz23OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. The paper argues that showing an interpretability metric is robust and reproducible doesn’t really establish that it measures what we think it does.
  2. They use Lstruct, which measures whether SAE features in PLMs co-activate across residues that are in 3D contact, as the central metric.
  3. They first show that Lstruct consistently favours masked models over causal models, and this difference survives several conventional robustness checks.
  4. They then show that a higher Lstruct isn’t specific evidence of structural organisation learned from native residue order (retraining on sequences with residue order shuffled raises Lstruct in 52/54 cells).
  5. They then show that random networks and simple amino-acid indicators also score (surprisingly) well and at the tested depths and primary training budget, trained masked models don’t clearly outperform random initialization (cysteine indicator beats every learned feature in 32/54 comparisons under a shared scoring convention).
  6. They finally show that by changing the minimum sequence separation you can reverse the model ranking, and extend the control framework to five other published or adapted metrics, with different outcomes across the checks.
Critical strengths
  1. Interesting core result against the metric a. A metric that passes a wide range of conventional robustness checks (model seeds, prediction matching, dictionary seeds, frozen random dictionaries, a dictionary-free analysis) still goes up in models that never saw native residue order (52/54 cells). This is pretty striking and good evidence that robustness doesn’t equal validity, and it’s a useful warning for anyone using similar metrics. b. Paired models sharing init, data and batch order, seeds as the unit of inference, and fold-disjoint fitting/scoring for the main Lstruct analysis make this pretty tight and the results convincing.

  2. Nice reusable framework a. The order-destroyed corpus is a principled adaptation of Sinha et al. to proteins. b. Liked the pairing with random-init and no-model baselines and showing they catch different failures (e.g ρ drops after order-destroyed training but scores higher at random init for the masked models), think this is a valuable technique that others should adopt

Critical weaknesses
  1. Lack of explanation why Lstruct goes up a. Paper says the mechanism is unresolved (and it does run a good number of diagnostics) but none of them quite pin it down -there could have been more of an exploration into this (e.g., the reconstruction-matched comparison they propose themselves, since order-destroyed SAEs reconstruct better in all 54 cells) b. This matters a lot as the specificity claim stands either way, but without a mechanism it’s hard to say whether the native masked–causal difference is the same artefact or how the metric could be fixed.

  2. Do the conclusions generalise? a. The model family is limited. b. The extended-budget check helps, the order-destroyed rise persists at 32× the training budget ( though this is only one seed). The native masked model’s Lstruct also rises above the observed random-init range at blocks 11 and 14, so the random-init comparison seems budget-dependent? c. The random-init and no-model checks avoid retraining the pLM, so applying them to something like ESM-2-8M seems like a useful next test. d. I think the additional-metric audit was useful though its conclusions probably needs to account for the adaptations and each metric’s intended interpretation (this matters because the paper pitches a general framework so it’s important to know whether these failures hold for the models these metrics are normally applied to).

Questions for authors
  1. For the annotation-based metrics (Concept-F1, single-latent F1, SAEBench etc ) what intended interpretation requires their scores to decrease after order destruction, given that composition is preserved? + how does the main-text claim that “every metric failed at least one applicable check” fit with App K’s decision not to impose pass/fail labels?
2 comments
AuthorsOpenReview ↗

Reply for reviewer tz23 (1/2)

Thank you for the careful review. We ran new analyses for each point you raised. We label your Critical Weaknesses W1a–W2d and your Question Q1, and answer in order, in two parts: W1a, W1b, W2a and W2c here; W2b, W2d and Q1 in part 2.

W1a (why LstructL_{\text{struct}} goes up). Thank you for raising the open mechanism. We ran the reconstruction-matched comparison you point to, and a composition null. Reconstruction quality does not explain the increase. A stratified permutation null by amino acid, and in causal models by amino acid and position, removes most of it.

  • Reconstruction. The order-destroyed model's dictionary reconstructs better. We stopped each order-destroyed dictionary when its held-out fit matched the native one, and also compared it with native dictionaries stopped at the same training step. The increase remains under both (blocks 11, 14 and 18; masked 3/3 seeds, p ≤ 0.049; causal 3/3, p < 0.001).
  • Composition. LstructL_{\text{struct}} compares each feature with a null in which its activations are shuffled among all residues of a protein, keeping the contacts fixed. The stratified null shuffles them only among residues of the same amino acid (the amino-acid null), or of the same amino acid in the same fifth of the protein's length (the amino acid × position null).
    • Masked: the amino-acid null removes the increase (our dictionaries: +0.0139 to +0.0009, 1 of 3 seeds positive, p = 0.80; raw residual stream: +0.0041 to −0.0005, 0 of 3, p = 0.16).
    • Causal: most of the increase survives the amino-acid null (our dictionaries: +0.0288 to +0.0218, p < 0.001; raw residual stream: +0.0217 to +0.0179, p = 0.004). The amino acid × position null reduces it to +0.0013 on the raw residual stream (p = 0.13); results on further held-out partitions will be in the revision.
    • Amino-acid identity alone predicts more of the activations of order-destroyed models than of native ones (3 of 3 seeds in both objectives, p ≤ 0.004).
  • Check on the null (positive control). We planted 289 synthetic features defined only by the contact graph: each is on, in every protein, at one residue and at its contacts (Cα within 8 Å, at least 12 apart in sequence), and off elsewhere. Under the amino acid × position null they keep a median of 66% of their score (10th–90th percentile 64–68%), while the order-destroyed increase largely disappears. So the null removes about a third of a known contact signal but most of the increase; the revised appendix will give the full construction.
  • Real contacts. Scored against an unrelated protein's structure (same length, another SCOPe fold; our dictionaries, 136 held-out domains), the causal increase falls from +0.026 to +0.004 (p = 0.020) and the masked one from +0.016 to +0.005 (p = 0.072). With the stratified null, this points to residue identity and position acting through real contacts.

We will add these results to the revision as a new Results subsection.

W1b (same artefact? a fix?).

  • Same artefact: largely. The amino-acid null removes about two thirds of the native masked–causal gap, and the amino acid × position null 82%. The null is an analysis of the response, not a corrected metric: randomly initialised networks still score above trained ones under it (3/3 seeds in both objectives, p ≤ 0.009), so even a composition-controlled version needs the three controls.

W2a and W2c (model family; ESM-2-8M). We ran the random-initialisation and no-model checks on ESM-2-8M and ESM-2-650M, at layers fixed in advance as the nearest to the paper's diagnostic depths (8M: 2–4 of 6; 650M: 13, 17, 21 of 33). With a dictionary, p-values compare one released model (three dictionaries) with three randomly initialised networks; without one, the networks with the single trained value.

  • Without a dictionary, trained ESM-2-650M scores above random weights at all three layers (p ≤ 0.023); at 8M the result is mixed (above at layer 4, at the random level at layers 2–3).
  • With our dictionary recipe, random weights score above the trained model at 5 of 6 layers (p ≤ 0.011), but that recipe fits trained ESM-2 poorly (at 8M, explained variance 0.42–0.58 against 0.96–0.98 for random weights). With dictionaries trained to fit (recipe chosen on fit alone, before scoring), they still do at 4 of 6 layers (p ≤ 0.004); the cysteine indicator then beats the best latent in 5 of 9 cells at 8M and none at 650M.
  • With published dictionaries (InterPLM; InterProt at its released layers 12, 16, 20), the cysteine indicator beats no latent at either size.
  • Under the amino-acid null, the random-weight score falls to about zero. Without a dictionary and with dictionaries trained to fit, trained ESM-2 scores higher at all six layers (p ≤ 0.003); with our recipe it does not (below random at 650M layers 13 and 21, p ≤ 0.005).
  • The order-destroyed check needs retraining; we ran it in new long-trained models (W2b).

So on released ESM-2 the result depends on the measurement step.

AuthorsOpenReview ↗

Reply for reviewer tz23 (2/2)

W2b (training budget). Partly. At 32× the budget (one seed) the native masked model scores above every random-initialisation draw at blocks 11 and 14, though not at 18, so that comparison shifts with budget; the order-destroyed increase persists in all six cells. We will state this in the revision. Much longer training removes neither the rise nor the random-weight result with dictionaries. In three new ESM-2-8M-architecture seed pairs (native and order-destroyed twins; about 240 times the paper's primary budget per parameter), LstructL_{\text{struct}} still rises after order destruction (3 of 3 pairs, p = 0.012, our dictionary recipe; with reconstruction matched, 3 of 3 but not significantly, p = 0.053; without a dictionary, not significant), and random weights still score above the trained models at layers 2 and 3 of the three tested (2–4 of 6; not at layer 4), with our recipe and with dictionaries trained to fit.

W2d and Q1 (additional metrics). Thank you for pointing out the adaptations and intended interpretations. We now compute the metrics with their authors' own functions, unchanged, wherever code exists (Adams et al., SAEBench, Li et al., InterPLM's concept-F1), with our labels, and the controls separate the metrics.

  • Whether a decrease is required depends on the label. No source paper tested its metric on shuffle-trained models, so none says how its score should respond. Every model is evaluated on native proteins with true annotations, and only the training data were shuffled, so a fall is expected only where the label depends on residue order:

    • For whole-protein labels (SCOPe class, fold, family; used by Adams et al. and SAEBench), composition alone can carry the signal, so a lack of decrease does not count against the metric; only the no-model and random-initialisation checks apply.
    • For residue-level labels that depend on sequence neighbours (secondary structure, burial), a decrease is expected, and §3.6's probes for these largely fall to random-initialisation level.
    • Li et al.'s statistic uses no labels: it asks whether co-firing features have similar decoder directions, which need not depend on token order, so no fall is expected and only the random-initialisation check applies. dstructd_{\text{struct}}, like LstructL_{\text{struct}}, reads 3D clustering of feature activations, so a fall is expected; InterPLM uses it to select features, not to compare models. SAEBench's authors designed their metrics to compare SAEs on one model; the order-destroyed comparison is our addition.
  • What the checks find. The revised main text will replace "every metric failed at least one applicable check" with this per-metric reading (our dictionary recipe, three seeds, SCOPe labels). We re-ran each metric at SAE sparsity kk = 32, 64, 128 and 256 active latents per residue (the paper used 256):

    • dstructd_{\text{struct}} (re-implemented from InterPLM's Methods): as §3.7 reports, it scores above the random-initialisation and no-model baselines yet rises after causal order destruction, at every kk and with reconstruction matched (+0.254, p = 0.001). Its causal rise in latents passing is the same with an unrelated protein's structure (+0.209 vs +0.212): with no sequence-separation rule, it favours features smooth along the chain.
    • Concept-F1 (InterPLM's functions): falls after causal order destruction at every kk. In masked models their reported mean rises because their code leaves out concepts with no test hit; over all concepts it does not.
    • Other metrics. In causal models single-latent F1 falls on all checks at every kk. SAEBench never moves against the intended direction at any kk, pooling the first 128 residues or the whole protein. With Li et al.'s firing threshold their clustering does not finish in 45 of 216 cells; with activation > 0, ρ\rho's direction after causal order destruction depends on the window (per residue it falls at every kk; over 8- or 32-residue blocks it rises at k≤128k \le 128).
    • Revised claims: single-latent F1 (developed by Adams et al. on ESM-2-650M) scores at chance in our masked models; we draw no conclusion from Li et al.'s statistic (developed on a text model, Gemma-2-2B).
  • When the model has learned order, the annotation metrics fall. In our long-trained pairs Adams et al.'s metric falls after order destruction in 6 of 6 and SAEBench's in 5 of 6 (one tie; three seed pairs, two layers; our recipe), and their native models score above a composition baseline (SAEBench only just), while LstructL_{\text{struct}} still rises in the same models. So SAEBench's mostly non-significant changes at 42M do not mean it is insensitive to order. With dictionaries trained to fit both fall in 6 of 6 (SAEBench's in 5 of 6 with reconstruction matched).

The revised Table 16 will give every metric-by-check outcome at each kk. All new results and their code will be in the revised PDF and the anonymised supplement, to be uploaded during the discussion period.

Reviewer_y3Y4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks the question that even when a relation-referenced interpretability metric can be reproducible across seeds, dictionaries and analysis choices, can it still not measure the relation it is named for. Existing controls (weight-shuffled models, random initialization, random dictionaries) randomize the model and leave the data intact, so they ask whether training is needed but not whether the named relation is. The paper presents a control that intervenes on the training distribution instead: permute residues within every training sequence, keeping length and composition, so that no real structure can be learned, and check whether the score drops.

Critical strengths
  1. The design is careful at every layer. The two models differ only in the attention mask and the loss it induces; The shuffled corpus keeps the length and composition unchanged for each sequence and reuses each native model's initialization and batch order. The falsification criterion is specified before looking at the result. The experiments are rigorous.
  2. The paper is well written.
Critical weaknesses
  1. The shuffle removes order but keeps composition, and composition is the obvious suspect: random init matches the trained masked score (Table 1), the authors pointed out this confound and its fix but ran neither. A composition-matched null — permute only within amino-acid type — would address it and does not need new models.
  2. Everything is shown on one architecture at one scale with one masking rate. The random-init and no-model baselines need no retraining; running them on ESM-2 8M, the model InterPLM used, and on 650M would show whether any of this carries over to models people actually analyze.
Questions for authors
  1. Composition-matched null: how much of the masked–causal gap and of the shuffled-corpus rise survives?
  2. A cross-feature version of L_struct (feature A at i, feature B at j) would see the relational encoding §2.4 says the current one misses, and would not be inflated by residue identity. Have you tried it?
2 comments
AuthorsOpenReview ↗

Reply for reviewer y3Y4 (1/2)

Thank you for the review and your two specific suggestions, the composition-matched null and the ESM-2 checks. We ran both and report the results below. We label your Critical Weaknesses W1–W2 and Questions Q1–Q2, and answer in two parts: W1 and Q1 here; W2 and Q2 in part 2.

W1 and Q1 (composition-matched null). We ran the null you proposed as a stratified permutation null: activations are shuffled only among residues of the same amino acid in each protein (amino-acid null), or of the same amino acid in the same fifth of the protein's length (amino acid × position null). How much survives:

  • Masked–causal gap (native models): about a third. The amino-acid null reduces it from +0.0145 to +0.0053 (our dictionaries; still positive in all 3 seeds, p = 0.045).
  • Increase after order destruction, masked models: none. The amino-acid null removes it (raw residual stream, no dictionary: +0.0041 to −0.0005, below zero in all 3 seeds; our dictionaries show the same).
  • Increase after order destruction, causal models: most survives amino acid alone, little survives amino acid × position. On the raw residual stream, the amino-acid null reduces it only from +0.0217 to +0.0179 (positive in all 3 seeds), while the amino acid × position null reduces it to +0.0013 (still positive in all 3 seeds, but not significantly: p = 0.13).

A null that holds amino-acid identity fixed could also remove genuine contact signal, since contacting residues tend to share amino-acid types. To check this, we built synthetic features defined only by the contact graph: in each held-out protein, a feature is switched on at one residue (the centre) and at every residue in 3D contact with it, and off everywhere else. We built 289 such features so that every residue with at least three contacts is a centre in at least one feature (9,485 centres in 151 proteins), and scored them the same way as learned features. They keep a median of 66% of their score under the amino acid × position null (10th–90th percentile 64–68%). So the null leaves most of a known contact signal in place, while the order-destroyed increase largely disappears under it. This is not a full explanation: a small part of the increase remains, and the null also removes about a third of a known contact signal. Results for every partition and dictionary setting will be in a new Results subsection of the revision.

W1 and Q1 (real contacts). The increase also depends on the proteins' real contacts. Scored against an unrelated protein's structure (same length, another SCOPe fold; our dictionaries, on the 136 of 151 held-out domains that have one), the causal increase falls from +0.026 to +0.004 (p = 0.020) and the masked one from +0.016 to +0.005 (p = 0.072). With the stratified null, this points to residue identity and position acting through real contacts.

W1 (reconstruction matching). We also ran the reconstruction-matching control proposed in the paper's Discussion.

  • We stopped order-destroyed dictionaries when their held-out fit matched the native partner. We also compared against native dictionaries stopped at the same training step.
  • The increase remains in both objectives and both comparisons (masked 3/3, p = 0.049 and 0.035; causal 3/3, p < 0.001).
  • Without any dictionary the direction also holds (6/6 seed summaries; causal p = 0.002, masked p = 0.11).
AuthorsOpenReview ↗

Reply for reviewer y3Y4 (2/2)

W2 (ESM-2-8M and ESM-2-650M). We ran both checks with three randomly initialised networks of each size, built with ESM-2's own release code, at layers fixed in advance as the nearest to the paper's diagnostic depths (8M: layers 2–4 of 6; 650M: 13, 17, 21 of 33). At each layer the one released model, scored through three dictionaries with different seeds, is compared with the three random networks, each scored through its own dictionary (Welch's t-test, 3 vs 3); without a dictionary, the three random networks are compared with the single trained value (one-sample t-test). With one released model, the p-values cover variation across dictionary fits and random initialisations, not across trained models.

  • Without a dictionary: at 650M the trained model scores above the random networks at all three layers (p ≤ 0.023); at 8M it does at layer 4 and is at their level at layers 2 and 3.
  • With a dictionary, the random networks score above the trained model at most layers of both sizes. Our recipe fits trained ESM-2 poorly (explained variance 0.42–0.58, against 0.96–0.99 for random weights), but dictionaries trained to fit (above 0.8; recipe chosen on fit alone, before scoring) give the same result.
  • With published dictionaries (InterPLM; InterProt at its layers 12, 16, 20), the cysteine indicator beats no learned feature at either size.
DictionaryRandom vs trained, 8M (layers 2 / 3 / 4)Random vs trained, 650M (layers 13 / 17 / 21)Cysteine beats best latent, 8MCysteine beats best latent, 650M
None (residual stream)n.s. / n.s. / ↓↓ / ↓ / ↓NANA
Our recipe↑ / ↑ / ↓↑ / ↑ / ↑7 of 90 of 9
Trained to fit↑ / ↑ / ↓↑ / n.s. / ↑5 of 90 of 9
Published (InterPLM, InterProt)NANA0 of 30 of 3

↑ / ↓ = random weights score significantly above / below the trained model at that layer (p < 0.05); n.s. = no significant difference. Cysteine: cells in which the cysteine indicator beats the best latent (3 layers × 3 dictionary seeds; 3 layers × 1 released dictionary for published ones). NA = not applicable: no latents without a dictionary; no published dictionaries for random weights.

So on released ESM-2 the comparison depends on the dictionary step, and a better fit does not remove it. The order-destroyed check would need retraining ESM-2. All layers and dictionary settings will be in the revised appendix.

Q2 (cross-feature LstructL_{\text{struct}}). Thank you; this is a natural extension, and it would capture the relational encoding that §2.4 says the current statistic misses. We will add it to the future work of the revision. One point for its design: different features at ii and jj would not by themselves remove residue-identity effects, since two different amino-acid indicators can also be enriched across contacts, so such a statistic would need the same three controls. The pairwise contact probes of §3.6, which read pairs of residue representations, address a related question.

All new results and their code will be in the revised PDF and the anonymised supplement, which we will upload during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

Summary of the revision

We thank the reviewers for their careful and constructive comments. The reviewers valued the tightly controlled paired design, the finding that a metric which passes the usual robustness checks can still rise when the relation it is named for is removed, and the combination of training-distribution, random-initialisation and no-model controls as a technique others can use.

We have undertaken the analyses that the reviewers requested or suggested. The core results of the submission are unchanged: LstructL_{\text{struct}} rises after order destruction in 52 of 54 cells, and the random-initialisation and no-model results, the cysteine indicator, the reversal with sequence separation and the reference readouts are as submitted. The additional analyses explain why the scores rise, test how far the results extend across model scales and training budgets, and clarify which link between a score and its interpretation each control tests.

  1. Why LstructL_{\text{struct}} rises (new §3.3, for y3Y4 W1 and Q1, tz23 W1a, 17LQ W2 and Q2). Matching SAE reconstruction quality leaves the LstructL_{\text{struct}} rise in place. However, a permutation null stratified by amino acid removes the rise in masked models, and a null stratified by amino acid and position removes most of it in causal models. The amino acid × position null keeps about two thirds of the score of planted contact features, which mark only true contacts, so it leaves most genuine contact signal in place. Scoring each protein against an unrelated structure also removes most of the rise, so the rise depends mostly on the proteins' real contacts. Structural probes on the same representations fall to or near the random-initialisation level (Appendix L). Together these point to residue identity and position acting through real contacts.
  2. Released and longer-trained models (new §3.6, for y3Y4 W2 and tz23 W2a–c). Without a dictionary, released ESM-2-650M scores above randomly initialised copies at all three layers tested. With a dictionary, even one trained to fit the released models, the random copies score higher at most layers of both ESM-2-8M and ESM-2-650M, showing that the outcome depends on the dictionary step. In three ESM-2-8M-architecture seed pairs trained about 240 times longer than our primary budget, LstructL_{\text{struct}} still rises after order destruction (3 of 3 pairs, p=0.012p = 0.012), while single-latent F1, concept-F1 and the SAEBench probe fall.
  3. Metric comparisons (§3.9, rewritten, for tz23 W2d and Q1, 17LQ W3 and Q3). Each metric is now computed with its authors' released code, using SCOPe, secondary-structure and solvent-accessibility labels for the proteins. Two settings defined for text models are translated to proteins, with the literal settings reported beside them (Appendix K). InterPLM's structural score has no released code and is re-implemented from their Methods. We reframe the section around what the controls show for each metric, in place of our original summary that every metric failed at least one applicable check. The annotation F1 metrics fall where the model has learned order. InterPLM's structural score rises in causal models, and the controls show that it does so through features that vary smoothly along the chain. Concept-F1's reported mean rises in masked models because the authors' reporting rule leaves out concepts with no test hit. ρ\rho's direction depends on the window. Counted per residue it falls after causal order destruction, and counted over blocks of 8 or 32 residues it rises at k≤128k \le 128, but most of that rise remains when residue positions are shuffled within each protein. We therefore draw no conclusion from it in the 42M models.

Reading the revision. Our earlier replies cite the initial submission. Its §3.5, §3.6 and §3.7 are now §3.7, §3.8 and §3.9 respectively, and the tables that our replies cite (8, 11, 14 and 16) keep their numbers. The readouts after order destruction, which our replies to 17LQ (Q1) and tz23 (W2d) cite under §3.6, are in Appendix L. The anonymised supplement adds a folder, additional_analyses/, whose README maps every new table and Appendix L paragraph to its code and per-cell results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.