- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The presented paper tests whether reproducible interpretability scores support their intended biological interpretation or not. In paired masked and causal pLMs the structural co-activation metric L_struct consistently favors masked models. Yet training on residue-permuted sequences increases the score in 52/54 comparisons. Randomly initialized models and amino-acid indicators provide further counter-examples.
Critical strengths
-
The paired training design controls architecture, initialization, corpus, and batch order. Replicated re-training interventions provide substantially stronger evidence than another comparison between unrelated pre-trained models.
-
The combination of shuffled training, random initialization, and deterministic indicators clearly demonstrates that a high structural co-activation score need not establish learning from native residue order. Sensitivity analyses address several straightforward implementation explanations.
Critical weaknesses
-
The conceptual novelty is limited. Shuffled pre-training already exposed problems with syntactic probes; random-model failures of SAE metrics were already demonstrated, where scores similar to those from trained models were reported. The main advance here is the controlled protein-specific counter-example and the combination of controls.
-
The central counter-example is stronger than the explanation. Shuffling preserves composition, changes the evaluation distribution relative to training, and improves SAE reconstruction in every matched cell; reconstruction improvement correlates with the score increase. The limited dictionary-free checks help, but the paper does not establish which preserved or altered property drives the result. This supports the narrow conclusion about score specificity, while leaving the underlying failure unresolved.
-
Broader metric failure claim is insufficiently justified. Composition and taxonomy information can survive order destruction, so annotation F1 or decoder-geometry agreement need not decrease unless their intended interpretation specifically requires native order. Several implementations and label sets differ from the original evaluations. Moreover, no statistically detectable decrease with three seeds does not establish insensitivity. These tests support metric-dependent responses more clearly than general invalidity.
Questions for authors
-
Do independent structural readouts decrease after shuffled training, particularly separation-matched contact probes? Report both raw representations and SAE latents.
-
Does the score increase persist after matching SAE reconstruction quality? Can a residue-conditioned permutation null separate biochemical clustering from contextual grouping?
-
For each additional metric, what precise published interpretation requires its score to decrease under this intervention? Please distinguish significant increases, inconclusive changes, and baseline failures, using common concept sets.
-
Why does the exact sign test treat the 6 objective-by-seed summaries as independent when paired objectives share initialization and training inputs? Report inference accounting for these three paired seed clusters.
6 comments
Second Reply to Reviewer 17LQ
W2 and Q2 (mechanism, matched reconstruction, and residue-conditioned null). Thank you for raising both points. The new analyses below show that the increase survives matched reconstruction, and that a residue-conditioned permutation null attributes most of it to residue identity and position.
- Reconstruction does not explain the increase (Q2, first part). The order-destroyed model's dictionary reconstructs better. We have now run the matching proposed in the paper's Discussion.
- We stopped each order-destroyed dictionary at the first step where its held-out explained variance reached that of the native model's dictionary. still rises (blocks 11, 14 and 18): masked 3/3 seeds, +0.0110, p = 0.049; causal 3/3, +0.0275, p < 0.001.
- It also rises against native dictionaries stopped at the same training step (masked 3/3, p = 0.035; causal 3/3, p < 0.001), and on the raw residual stream with no dictionary (causal +0.0217, p = 0.002; masked +0.0041, p = 0.11; 3/3 seeds positive in each).
- Sparsity is another free analysis choice that shifts metric verdicts, reinforcing §3.5. The paper fixed active latents. The causal rise reproduces at and and without a dictionary (p ≤ 0.038; masked same direction, n.s.). At the causal direction reverses (0/3 seeds, p = 0.009; repeat p = 0.055) while the masked rise keeps its direction (3/3, p = 0.15). Because four of the five published metrics, computed with their authors' functions where code exists, also change verdict on at least one check across (all but single-latent F1; W3 and Q3), controls must be run at the exact dictionary settings used in an evaluation; we will add the full -grid to the revision.
- Composition: a stratified permutation null removes the masked increase and most of the causal one (Q2, second part). compares each feature with a null in which its activations are shuffled among all residues of a protein, keeping the contacts fixed. The stratified null shuffles them only among residues of the same amino acid (the amino-acid null), or of the same amino acid in the same fifth of the protein's length (the amino acid × position null).
- Masked: the amino-acid null removes the increase. With our dictionaries it goes from +0.0139 to +0.0009 (p = 0.80), and 1 of 3 seeds stays positive. On the raw residual stream, with no dictionary, it goes from +0.0041 to −0.0005 (p = 0.16), and 0 of 3 seeds stay positive.
- Causal: the amino-acid null leaves most of the increase (raw residual stream: +0.0217 to +0.0179, 3 of 3 seeds still positive, p = 0.004; our dictionaries: +0.0288 to +0.0218, p < 0.001). The amino acid × position null removes the rest with our dictionaries (+0.0288 to −0.0003, 0 of 3 seeds positive, p = 0.18) and reduces it to +0.0013 on the raw residual stream (p = 0.13); results on further held-out partitions will be in the revision.
- Planted contact features (289 synthetic features; in every protein each is on at one residue and at its contacts, off elsewhere; construction in the revised appendix) keep a median of 66% of their score under the amino acid × position null (10th–90th percentile 64–68%), while the order-destroyed increase largely disappears. So the null removes about a third of a known contact signal but most of the increase.
- Amino-acid identity alone predicts more of the activations of order-destroyed models than of native ones (share of held-out variance, raw residual stream, native to order-destroyed: masked 0.74 to 0.97; causal 0.16 to 0.45; 3 of 3 seeds, p ≤ 0.004).
- Real contacts: the increase depends on them (W2). Scored against an unrelated protein's structure (same length, another SCOPe fold; 136 of the 151 held-out domains have one), the causal increase falls from +0.026 to +0.004 (p = 0.020) and the masked one from +0.016 to +0.005 (p = 0.072). Because the null ignores sequence separation, it penalises features that vary smoothly along the chain. Referencing each score to its value against unrelated structures removes this bias, and the increase stays positive (masked +0.011, p = 0.021; causal +0.022, p = 0.004; on 400 dictionary-fitting domains +0.009 and +0.012, p ≤ 0.043). With the stratified null, this points to residue identity and position acting through real contacts.
- Shift in evaluation distribution: it does not explain the increase on its own. The order-destroyed models are scored on native proteins they never saw, as in any training-distribution control (Sinha et al.'s shuffle-trained models also show much higher perplexity on natural text). But the mismatch does not raise every readout: on the same inputs, structure probes of the order-destroyed models fall to the random-initialisation level (Q1). And with less mismatch, the block-shuffled models of §2.2 (16-residue segments reordered, order kept within each) still raise in all 9 causal cells and 6 of 9 masked cells (§3.2).
Third Reply To Reviewer 17LQ
Q1 (structural readouts, raw and SAE). We ran these readouts on the order-destroyed models. On both the raw residual stream and SAE latents, the structural readouts of §3.6 fall back to the random-initialisation level after order destruction. Below, "layer" means the paper's blocks 11, 14 and 18.
- Residue-level probes (helix, strand, burial; the protocol of Table 14, unchanged).
- Causal: order-destroyed models lose 86–97% of the native models' gain over random initialisation (layer means; 65–113% across the 9 seed-by-layer cells per label).
- Masked: order-destroyed models lose 95–106% at layer 18. At layers 11 and 14 the native masked models are themselves barely above random initialisation on these probes (Appendix I), so there is little to lose.
- Pairwise contact probe (§3.6, Appendix I): a logistic-regression probe on pairs of residue representations, in which each contact is matched with a non-contact from the same protein at the same sequence separation. We say the probe falls after order destruction when the native model scores higher than the order-destroyed one: the bootstrap interval for native minus order-destroyed lies above 0 in at least 2 of 3 seeds (rule fixed before the runs).
- Raw residual stream: order-destroyed models are at the random-initialisation level in all 6 objective-by-layer cells. By the rule, the probe falls at masked layer 18 and causal layers 11 and 18.
- SAE latents (our dictionary recipe): the same picture. Order-destroyed models are at the random-initialisation floor in all 6 of 6 cells, and by the rule the probe falls in 4 of 6 cells: masked layer 18 and causal layers 11, 14 and 18 (native AUROC 0.531–0.565 against random initialisation 0.527–0.538).
- Contrast with . On the exact representations where rises after order destruction, supervised structural probes drop to the random-initialisation floor. (Note that at this training budget, native contact probes themselves only reach the ~0.59 AUROC of a model-free amino-acid-pair and sequence-distance baseline on a different split—yet unlike , they never rise significantly when sequence order is removed.)
First Reply To Reviewer 17LQ
Thank you for a precise review. We ran the analyses you asked for and report them below. We label your Critical Weaknesses W1–W3 and Questions Q1–Q4, and answer in five parts, pairing overlapping points: W1 here; W2 with Q2 in part 2; Q1 in part 3; W3 with Q3 in part 4; Q4 in part 5.
W1 (novelty). Shuffled pre-training (Sinha et al., 2021), random-model baselines (Heap et al., 2025) and label randomisation (Adebayo et al., 2018, which we will cite) are precedents. We differ in two ways: the intervention removes one targeted relation (native residue order) while keeping the properties that could otherwise produce the score (length and composition), and the criterion is directional (a measure of a relation learned from data should not rise when that relation is removed).