- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes FalsifyVC, a framework for testing whether a virtual-cell predictor’s score improvement over a declared comparator strictly exceeds a registered margin with sufficiently high probability. Its organizing concept is comparator authority: the information available for selecting a comparator and the selection rule. The evaluated authorities are a fixed training-only comparator, a training-only per-target selector, and a per-target envelope chosen using held-out responses. For each complete training run, the framework averages delta-correlation scores over targets and measures the excess over the comparator; when several environments are assessed jointly, it takes the minimum excess across those environments. Standard one-sided exact binomial tests are then applied to the number of runs strictly exceeding the margin, yielding supported, refuted, or inconclusive verdicts for the specified reliability claim.
The initial campaign contains 60 training runs of GEARS, STATE, and CPA on two Replogle screens. Follow-ups evaluate GEARS with reference preprocessing, matched training cells, and twelve fresh target splits per screen, as well as corrected-label Adamson experiments with thirty training seeds on one split and thirty additional fresh splits. GEARS never clears the margin against the primary training-selected comparator in these follow-ups (Appendices N and O). A STATE follow-up supplies training-derived gene features and gives 0/10 successes against each evaluated comparator on each Replogle screen (Appendix P). Additional analyses examine weak-selector effects, envelope selection optimism, coverage, control-reference sensitivity, and operating characteristics. In the Norman positive control, all four comparator specifications are registered before the run-level scores: the predictor is supported against the original fixed comparator and envelope, but refuted against the additive comparator and the envelope enlarged to include it (Appendix R). The contribution is an explicit comparison specification and strengthened conditional empirical evidence, rather than a new statistical test or a universal negative claim about virtual-cell prediction.
Critical strengths
- The follow-up experiments substantially strengthen the empirical negative result by addressing plausible alternative explanations. Appendix O gives GEARS reference preprocessing, exactly matched training cells, and twelve fresh target splits per Replogle screen. On the covered targets, all runs remain below the training-selected comparator, with mean excesses of −0.082 and −0.097 and best-run excesses of −0.069 and −0.058 for K562 and RPE1, respectively (Table O.1). Corrected-label Adamson experiments similarly give 0/30 successes against the primary training-selected comparator, both across training seeds on one split and across fresh splits (Tables N.1 and N.2). These results are stronger than a deficit on one raw-count split, and their negative direction does not depend on the particular positive margin. Appendix P also supplies training-derived gene features to address STATE’s original untrained unseen-target representation, with the effective coverage requiring the clarification below. The combined evidence is a useful contribution to a contested empirical question, even though the statistical tools are standard.
- The diagnostics make the comparison interpretable and examine important failure modes of evaluation. Comparisons against training-selected ridge and a stronger training-only selector show that GEARS’s positive mean excess over the original selector does not persist against these stronger comparators (Appendix I). Choosing the envelope member on disjoint cells substantially reduces its measured advantage over the fixed comparator; this is an informative diagnostic, although finite-sample member-selection errors also affect the comparison (Appendices J and S). The paper distinguishes run reliability from mean improvement and reports multiplicity and coverage sensitivities. Its controls include thirty random-feature runs on Norman and thirty training-cell bootstrap refits per setting for the near-margin linear predictor (Appendices R and Q). These controls demonstrate attainable support and behavior near the margin under their respective run-generating procedures; they do not establish calibration or power for arbitrary neural training pipelines.
Critical weaknesses
- The conceptual contribution is modest, and authority indexing has limited demonstrated incremental decision value. The fixed/selector/oracle distinction, margin-based run comparisons, and exact binomial inference have established precedents, which the paper acknowledges (Section 2 and Appendix V). With the same score and target set, an envelope adds no new refutation once a constituent fixed comparator already refutes. The original selector was already substantially weaker on the development split (Appendix E), and the stronger selector does not materially improve on training-selected ridge (Appendix I). The Norman control contrasts comparator specifications with and without an explanatory additive baseline; it does not isolate an effect of changing authority while keeping the comparator family fixed. Consequently, I view the strongest contribution as empirical validation and diagnostic specification, rather than a new inferential principle. This limits the strength of the acceptance case, although it does not negate the empirical contribution.
- The scientific interpretation remains conditional on the tested protocols and metrics. The initial campaign uses one fixed, untuned recipe per model; later follow-ups change preprocessing or perturbation features while retaining specified training budgets (Appendices B, O, and P). The primary score is mean per-target delta correlation, and the margin is based on variability over six raw-count GEARS development seeds, rather than biological utility (Appendix D). The authors identify the initial CPA and one-hot STATE tests as outside their supported regimes and explicitly do not rest the negative conclusion on them. Reference-preprocessed GEARS and feature-equipped STATE substantially improve the evidence, but do not establish performance across all other recipes, biological contexts, or endpoints. The authors acknowledge these restrictions. They limit the breadth and practical interpretation of the findings, rather than demonstrating that the reported conditional refutations are invalid. The expression-scale chronology and STATE coverage details below would further improve auditability.
Questions for authors
- Expression-scale chronology. Section 5 and Table T.1 state that per-model expression scales were fixed after 17 runs. Appendix B and Table B.1 state that the retained runs use a fixed configuration and differ by seed. Please connect these statements explicitly: did the scale decision concern only scoring fixed predictions, or the choice of complete training-and-scoring specifications? Which components of the two raw/log candidate specifications were fixed before the relevant outcomes were observed? A short anonymized protocol chronology or registration excerpt would make the application of Corollary 1 easier to audit. I am not asserting that runs were selectively excluded or that configurations were mixed; neither is established by the manuscript.
- STATE feature coverage and validation. Appendix P already states that targets outside the gene panel receive zero-vector features. Please specify the target counts and membership underlying Table P.1’s covered and filled arms, including whether zero-feature targets are counted as covered. Are the results unchanged when restricted to targets with training-derived gene features? Please also clarify how the preprocessing and 4,000-step budget relate to the intended configuration of the tested release and summarize any existing validation/checkpoint evidence supporting this configuration. This is a request to clarify the supported-regime interpretation, not an assertion that the model was undertrained.
4 comments
Author Response to Reviewer L1u4 (Part 2, Continued)
Selector strength and the margin. We agree that the stronger training-only selector does not improve on training-selected ridge (Table I.2). Its role is to show that ridge is not itself a weak comparator, so GEARS's positive excess over the registered selector does not survive a stronger selector. The margin is calibrated on run-to-run variability across six GEARS development seeds (Appendix D) and does not measure biological utility. Our conclusions remain conditional on mean per-target delta correlation and the tested recipes.
Zero-feature targets (all other frozen test targets carry nonzero features, Appendix P).
K562 (33 of 404). ABCG1, BGLAP, CCDC144A, CDAN1, CENPI, CPAMD8, DNLZ, EIF3CL, ESPN, F8A1, FRG2, HAPLN2, HIST1H2BB, HIST2H3A, KRTAP4-2, LCE1C, LPA, MRPL53, NOMO3, NPB, NRF1, OR4F4, RNF103, SGPP1, SIGLEC14, SLC16A5, TREX2, TULP1, ZNF133, ZNF236, ZNF253, ZNF674, ZNF718.
RPE1 (55 of 441). CA5A, CCDC144A, DDN, DEPDC5, DMRTA2, DPY19L2, EPS8L1, FAM102B, FAM229A, FOLR3, FOXL2, GAB2, HCRTR1, HIST1H2BN, HYPK, KRTAP4-2, LCE1E, LCN10, MED18, MED9, MEIS3, MEMO1, NBPF15, NFKBIE, NOMO1, NOP9, NPB, OR2T5, OR4F4, OR7A17, OTX1, PLA2G10, PTCD1, RGPD6, RNF123, RPTOR, S100A1, SIGLEC14, SLC16A5, SPDYE2, STAT5A, STAT5B, TBX1, TDGF1, TRAIP, TREML2, TUT1, WDR24, WDR7, YPEL1, ZBTB14, ZFP69B, ZNF133, ZNF284, ZNF492.
Reviewer response
Thank you for your explanations. I'll keep my initial score (3).
Thank you for taking the time to read our response and for your constructive feedback.
Author Response to Reviewer L1u4 (Part 1)
Thank you for these questions. The amendment changed scoring, not training recipes. Zero-feature targets count as covered. A bound preserves all six refutations on nonzero-feature targets at the registered margin, although exact restricted means remain unavailable.
Expression-scale chronology. The September 10 registration (Row 1 of Table T.1) fixed the metric, splits, comparator family, A1 member, coverage rule, δ, ρ, α and R=10. Table B.1 records the fixed training recipes and input scales. Row 2 discloses earlier test access from three one-epoch pipeline checks and model-free comparator scores, excluded from campaign counts.
Row 3, dated September 12 after seventeen completed campaign runs, concerns only the scoring of existing predictions. GEARS had been trained on raw counts and STATE and CPA on log(1+x) inputs, whereas the analysis loader read raw-count truth for all three because Row 1 had omitted the scoring dataset version. The amendment aligned truth, control subtraction and comparator fitting with each model's input scale. Existing predictions and training hyperparameters were retained, and no completed seed was rerun. The recorded criterion was agreement with independent wrapper scores, not score maximization.
Those seventeen runs were GEARS seed 1, STATE seeds 1 to 3 and CPA seeds 1 to 3 on K562, and GEARS seed 1, STATE seed 1 and CPA seeds 1 to 8 on RPE1. Their saved arrays are finite, and all sixty retained configurations match this scale assignment. These files corroborate the record without proving its historical timing. Across the four raw-or-log scoring combinations over the two screens, the K=0 tail 0.2510=9.54×10−7 lies below α/4. Corollary 1 covers fixed scoring choices, not adaptive training recipes. Appendix A and Table T.1 now separate these two timelines.
Coverage and membership. Both arms of Table P.1 score all 404 K562 and 441 RPE1 test targets (Appendix P). Covered means that a prediction exists, including for zero-feature targets. The input tensors carry nonzero training-derived features for 371 and 386 targets, the exact held-out genes present in each expression panel. The remaining 33 and 55 targets receive the control's zero-vector fallback and are listed in the companion comment below.
Nonzero-feature verdicts. Let N be the full target count, m the zero-feature count, Wall the full-target mean excess and cˉ0 the comparator's mean score on zero-feature targets. Since each model correlation is at least −1,
Wnonzero≤N−mNWall+m(1+cˉ0).
Retained per-target comparator scores, selection maps and linear-predictor scores determine cˉ0 for all six comparators and reproduce their stored full-target means. The largest upper bound over runs and comparators is −0.0383 on K562 and +0.0056 on RPE1, both below δ=0.0139. Thus K=0/10 against all six comparators on both nonzero-feature subsets, with no imputed model scores. Exact restricted means are unavailable because predictions survive for only six of the twenty runs and per-target model scores were not retained.
Release and training evidence. The fixed, untuned recipe uses arc-state 0.10.5/cell-load 0.10.3, log(1+x) raw counts on the full panel without library-size normalization, training-only ten-PC features, state_sm, 4,000 steps, batch size 2, cell sets of 32 and validation every 20 steps. In six surviving traces, runtime perturbation mappings match the input feature tensors. Each records 200 validation observations, finite losses, a 4,000-step terminal message and prediction from final.ckpt, without test-score checkpoint selection. Retry traces cannot be reverified for all twenty runs. Late validation loss still improves, so convergence and optimal release performance are not shown. The post-hoc 40,000-step follow-up uses the same recipe and five seeds per screen (Appendix P, Table P.2). Rescoring the six surviving 4,000-step prediction arrays reproduced their stored scores exactly. Minimum validation loss occurs between steps 30,400 and 38,600. All ten scores rise over the same-seed 4,000-step run, by 0.128 on K562 and 0.094 on RPE1 on average. Mean excess over registered A1 is −0.098 and −0.100, yet every comparator refutes on both screens (K=0/5 in both arms), rejecting success rates ≥0.75. The README's larger model and batch size remain untested.
Contribution. The A3 envelope adds no refutation once a constituent fixed comparator refutes on the same targets and score. Norman changes the comparator family rather than isolating authority. Our contribution is a comparison specification and conditional empirical evidence, not a new statistical principle.