- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a label-free diagnostic for how an input encoding restricts which measured contrasts a predictor can jointly produce, and applies it to an audit of a modified-siRNA graph neural network (GNN).
The diagnostic (Sec. 2, App. A10.1/A10.6). Given distinct raw endpoint states, a signed contrast design C ∈ R^{q×m}, and an encoder that partitions states into k equal-input classes via H ∈ {0,1}^{m×k}, the contrast vectors attainable by an unrestricted deterministic decoder are exactly col(CH) (Prop. 2.1). Forced-zero functionals form the left null space, of dimension q − rank(CH). Projecting the recorded contrast vector onto col(CH) gives an empirical squared-error floor for any such decoder. The rank is label-free; the floor is empirical. Corollary 2.2 gives interaction rank (a−1)(b−1) for a full anchored product panel.
The audited instance (Sec. 2, 7, 8). On a 15×11 antisense-by-sense sub-rectangle the authors extract from the Bramsen et al. (2009) screen (165 states, 140 rectangles), the GNN’s training-only node-feature mask zeroes 44 of 93 columns and merges 165 states into 90 classes, cutting rank(CH) from 140 to 72. No row of CH is zero, so no single rectangle is forced to cancel; the restriction is joint. The floor is 0.009980, which is 14.60% of the fitted interaction error 0.068335 and 14.63% of recorded contrast energy. A minimum of three restored chemistry columns recovers full rank (six distinct minimal triples; marginal gains 27+27+8 = 62 against 68 jointly, so non-additive).
Interventions. Refitting without the mask restores rank 140 and removes the floor entirely, but interaction MSE moves only 0.068335 → 0.068318, i.e. +0.000017, or 0.17% of the removed floor, with paired-seed changes in [−0.000176, +0.000262]. The restored columns remain zero on every training node. A pre-registered synthetic sweep (App. A10.11.2) tested the same question on constructed designs: the registered prediction that removing the class-equality restriction would lower out-of-subspace error below the floor failed (8/42 below, 34/42 above, median ratio 1.33), and total MSE was higher in 28/42 fits.
Negative control and breadth. A released MAVE-NN splicing predictor has an injective one-hot encoding, so rank(CH) = rank(C) = 1,986 and the floor is exactly zero; its 97 forced-zero functionals are design dependencies the measurements satisfy. A frozen breadth audit (App. A10.22) adds an expanded Bramsen panel (730 of 1,845 directions lost, floor 0.007565) and a GB1 binding panel under the same one-hot rule (floor zero).
Supporting audit material. Sections 3–8 add an activity-prediction evaluation on three cohorts, weighting and identity-hash conventions, provenance adjudication (299 unresolved primary/released label discrepancies), a proof that relation blocks absent from every training graph cannot change any training prediction (8,192 scalars), a bounded perturbation test, and preflight failures for two published modified-siRNA predictors, neither of which could be run.
Critical strengths
- A sound, discriminating diagnostic with a working negative control.
The paper separates what an encoding permits from what a fitted model achieves, and computes the first exactly: ranks by rational elimination on integer matrices, verified left-null witnesses, fitted contrast vectors checked to lie in col(CH) to 10⁻¹⁶, and the encoding partition verified on all 27,225 state pairs. Critically, it distinguishes encoding-induced restrictions, which the recorded contrasts violate and therefore pay for, from design-induced dependencies, which the measurements satisfy and which cost nothing. The MAVE-NN splicing instance returns rank(CH) = rank(C) = 1,986 and a floor of exactly zero, so the check does not fire automatically on a new panel.
Why it is consequential: This is the only part of the paper that could stand on its own, and it is correct. Without the negative control I would treat the positive floor as an artifact of the method; with it, the diagnostic is at least measuring something real about the declared encoders.
- Negative results are reported, including a failed pre-registered prediction.
The authors retract their own protocol’s claim that improvement is bounded by the removed floor, explain why refitting can also change in-span error, report that the pre-registered synthetic sweep’s registered prediction failed (34 of 42 nonzero-deficit fits above the floor), and state plainly that 85.40% of the GNN’s squared error lies inside the permitted contrast space.
Why it is consequential: It makes the reported numbers credible, so I read the nulls as evidence rather than as missing work. This is what keeps the paper near the threshold rather than well below it.
Critical weaknesses
- On the only panel where the floor is positive, the contribution reduces to a triviality, and the headline percentage is near-vacuous.
App. A10.11.1 proves the 68-dimensional left null space is spanned by duplicate-row differences: every restriction says that two rectangles the encoder cannot distinguish get equal predictions. No instance is shown where the forced-zero structure is anything else, which is the regime where Prop. 2.1 would earn its generality. Separately, the audited model’s 140 contrasts have SD 0.0016 against a measured 0.22, and its interaction MSE is 0.19% worse than a zero-interaction control. With the fitted contrast vector ≈ 0, the floor’s share of fitted error necessarily approaches its share of recorded contrast energy — and it does: 14.6038% against 14.63%. The abstract leads with the fitted-error figure, which therefore carries almost no information about the fitted model.
Why it is consequential: This is my main reason for a below-threshold score. It removes both the theoretical novelty and the empirical headline. What remains is that 85.4% of the error lies inside the permitted subspace with no identified cause.
- Every test of practical consequence returned null, and breadth is one lossy encoder.
Removing the mask eliminated the entire floor and changed MSE by +0.000017, with paired-seed changes straddling zero. The pre-registered synthetic sweep found total MSE higher for the unrestricted arm in 28 of 42 fits. The diagnostic-utility analysis reports a primary interval including zero. Meanwhile the positive-floor rows of Table 24 are one encoder on one study, the zero-floor rows are one released one-hot encoder, and both published modified-siRNA predictors failed their readiness gate, so no independent lossy encoder was ever tested.
Why it is consequential: A diagnostic is worth publishing if it either predicts something or is shown to matter somewhere. On these panels it does neither, and the authors say so. An independent lossy encoder with a positive floor, or a restoration experiment that decouples representability from training support and shows realizable gain, would change my assessment.
Questions for authors
- Is there a measured panel and encoder for which the forced-zero functionals are not spanned by duplicate-contrast equalities? If not, what does Prop. 2.1 buy over directly checking which rows of CH coincide?
- Given that the fitted contrast vector is near zero (SD 0.0016 against measured 0.22), what does the floor’s 14.6% share of fitted error tell a reader that its 14.63% share of recorded contrast energy does not? Would you consider leading with the data-only quantity?
- In the mask-restoration refit, the restored columns were zero on every training node. Can you run an arm in which restored chemistry also appears in training — e.g. by admitting duplexes that exercise those columns, or by the fixed-size replacement design used for MAVE-NN — so that representability and training support are actually decoupled?
- Given the failed pre-registered prediction in App. A10.11.2 and the near-zero refit gain, is there any setting in which the floor predicts post-restoration improvement? If not, how should a practitioner use it?
- Can the diagnostic be applied to at least one independent encoder with a lossy encoding, other than the audited mask? For instance, do standard tokenizations in published chemistry or sequence models merge measured states on a public panel?
- The antisense-marginal floor share is 38.1% against 14.6% for interactions, and the error decomposition places the deficit in main effects. Why does the title and abstract foreground the interaction floor?
- The AI use statement says AI tools were used throughout. Could you state more specifically which components (text, code, proofs, number checking) were AI-assisted, and what verification each received? This bears on how a reader should weigh the self-reported checks.
We wanted to thank you again for your review and to briefly highlight the new evidence addressing one of the forms of evidence you identified as potentially changing your assessment: an independent lossy encoder with a positive empirical floor.
Section 9, Appendix A10.23 and Table 24 report this example. Its 800 encoded contrast rows are nonzero and pairwise nonproportional, yet have rank 740, revealing 60 restrictions that duplicate-row checks miss. The empirical floor is 0.3150033, or 2.54% of recorded contrast energy.
We also completed the siRNA training-support experiment requested in Q3. Section 8 and Appendix A10.25 report its near-zero outcome and limitations.
Our point-by-point responses provide the details and supporting records. If you have an opportunity to revisit them before the discussion closes, we would appreciate your assessment of whether this evidence addresses your concerns. Thank you for your time.