acceptodds
Under review as a conference paper at ICLR 2027

FalsifyVC: Error-Controlled Reliability Testing of Virtual Cell Models across Comparator Authorities

Abstract

Virtual cell predictors are judged by held-out accuracy on unseen perturbations. Their added value depends on the comparator's selection information and rule, which existing reliability tests take as given. FalsifyVC indexes verdicts by this comparator authority, separating the reliability threshold, margin and run budget. Exact binomial tests count training runs whose excess over the comparator clears a registered margin. On two Replogle screens, no GEARS run clears the margin against any comparator across twelve fresh target splits with its reference preprocessing, or against fixed pseudobulk and a held-out per-target envelope on the registered split. STATE and CPA are refuted outside their supported regime, STATE also with training-data gene features. A training-only selector yields positive GEARS mean excess on the registered split but is weaker than training-selected ridge, and the excess vanishes on fresh splits. On an independent screen GEARS clears the margin in none of thirty runs, on one split or across thirty. A registered positive control clears the margin in all thirty runs and is supported, then refuted once an additive baseline is registered. The results refute the specified added-value claims under these conditions and show that the rule can support a large advantage over registered comparators.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
virtual cell models, perturbation prediction, evaluation methodology, error control, multiple comparisons, comparator choice, Perturb-seq

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_L1u4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FalsifyVC, a framework for testing whether a virtual-cell predictor’s score improvement over a declared comparator strictly exceeds a registered margin with sufficiently high probability. Its organizing concept is comparator authority: the information available for selecting a comparator and the selection rule. The evaluated authorities are a fixed training-only comparator, a training-only per-target selector, and a per-target envelope chosen using held-out responses. For each complete training run, the framework averages delta-correlation scores over targets and measures the excess over the comparator; when several environments are assessed jointly, it takes the minimum excess across those environments. Standard one-sided exact binomial tests are then applied to the number of runs strictly exceeding the margin, yielding supported, refuted, or inconclusive verdicts for the specified reliability claim.

The initial campaign contains 60 training runs of GEARS, STATE, and CPA on two Replogle screens. Follow-ups evaluate GEARS with reference preprocessing, matched training cells, and twelve fresh target splits per screen, as well as corrected-label Adamson experiments with thirty training seeds on one split and thirty additional fresh splits. GEARS never clears the margin against the primary training-selected comparator in these follow-ups (Appendices N and O). A STATE follow-up supplies training-derived gene features and gives 0/10 successes against each evaluated comparator on each Replogle screen (Appendix P). Additional analyses examine weak-selector effects, envelope selection optimism, coverage, control-reference sensitivity, and operating characteristics. In the Norman positive control, all four comparator specifications are registered before the run-level scores: the predictor is supported against the original fixed comparator and envelope, but refuted against the additive comparator and the envelope enlarged to include it (Appendix R). The contribution is an explicit comparison specification and strengthened conditional empirical evidence, rather than a new statistical test or a universal negative claim about virtual-cell prediction.

Critical strengths
  1. The follow-up experiments substantially strengthen the empirical negative result by addressing plausible alternative explanations. Appendix O gives GEARS reference preprocessing, exactly matched training cells, and twelve fresh target splits per Replogle screen. On the covered targets, all runs remain below the training-selected comparator, with mean excesses of −0.082 and −0.097 and best-run excesses of −0.069 and −0.058 for K562 and RPE1, respectively (Table O.1). Corrected-label Adamson experiments similarly give 0/30 successes against the primary training-selected comparator, both across training seeds on one split and across fresh splits (Tables N.1 and N.2). These results are stronger than a deficit on one raw-count split, and their negative direction does not depend on the particular positive margin. Appendix P also supplies training-derived gene features to address STATE’s original untrained unseen-target representation, with the effective coverage requiring the clarification below. The combined evidence is a useful contribution to a contested empirical question, even though the statistical tools are standard.
  2. The diagnostics make the comparison interpretable and examine important failure modes of evaluation. Comparisons against training-selected ridge and a stronger training-only selector show that GEARS’s positive mean excess over the original selector does not persist against these stronger comparators (Appendix I). Choosing the envelope member on disjoint cells substantially reduces its measured advantage over the fixed comparator; this is an informative diagnostic, although finite-sample member-selection errors also affect the comparison (Appendices J and S). The paper distinguishes run reliability from mean improvement and reports multiplicity and coverage sensitivities. Its controls include thirty random-feature runs on Norman and thirty training-cell bootstrap refits per setting for the near-margin linear predictor (Appendices R and Q). These controls demonstrate attainable support and behavior near the margin under their respective run-generating procedures; they do not establish calibration or power for arbitrary neural training pipelines.
Critical weaknesses
  1. The conceptual contribution is modest, and authority indexing has limited demonstrated incremental decision value. The fixed/selector/oracle distinction, margin-based run comparisons, and exact binomial inference have established precedents, which the paper acknowledges (Section 2 and Appendix V). With the same score and target set, an envelope adds no new refutation once a constituent fixed comparator already refutes. The original selector was already substantially weaker on the development split (Appendix E), and the stronger selector does not materially improve on training-selected ridge (Appendix I). The Norman control contrasts comparator specifications with and without an explanatory additive baseline; it does not isolate an effect of changing authority while keeping the comparator family fixed. Consequently, I view the strongest contribution as empirical validation and diagnostic specification, rather than a new inferential principle. This limits the strength of the acceptance case, although it does not negate the empirical contribution.
  2. The scientific interpretation remains conditional on the tested protocols and metrics. The initial campaign uses one fixed, untuned recipe per model; later follow-ups change preprocessing or perturbation features while retaining specified training budgets (Appendices B, O, and P). The primary score is mean per-target delta correlation, and the margin is based on variability over six raw-count GEARS development seeds, rather than biological utility (Appendix D). The authors identify the initial CPA and one-hot STATE tests as outside their supported regimes and explicitly do not rest the negative conclusion on them. Reference-preprocessed GEARS and feature-equipped STATE substantially improve the evidence, but do not establish performance across all other recipes, biological contexts, or endpoints. The authors acknowledge these restrictions. They limit the breadth and practical interpretation of the findings, rather than demonstrating that the reported conditional refutations are invalid. The expression-scale chronology and STATE coverage details below would further improve auditability.
Questions for authors
  1. Expression-scale chronology. Section 5 and Table T.1 state that per-model expression scales were fixed after 17 runs. Appendix B and Table B.1 state that the retained runs use a fixed configuration and differ by seed. Please connect these statements explicitly: did the scale decision concern only scoring fixed predictions, or the choice of complete training-and-scoring specifications? Which components of the two raw/log candidate specifications were fixed before the relevant outcomes were observed? A short anonymized protocol chronology or registration excerpt would make the application of Corollary 1 easier to audit. I am not asserting that runs were selectively excluded or that configurations were mixed; neither is established by the manuscript.
  2. STATE feature coverage and validation. Appendix P already states that targets outside the gene panel receive zero-vector features. Please specify the target counts and membership underlying Table P.1’s covered and filled arms, including whether zero-feature targets are counted as covered. Are the results unchanged when restricted to targets with training-derived gene features? Please also clarify how the preprocessing and 4,000-step budget relate to the intended configuration of the tested release and summarize any existing validation/checkpoint evidence supporting this configuration. This is a request to clarify the supported-regime interpretation, not an assertion that the model was undertrained.
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer L1u4 (Part 1)

Thank you for these questions. The amendment changed scoring, not training recipes. Zero-feature targets count as covered. A bound preserves all six refutations on nonzero-feature targets at the registered margin, although exact restricted means remain unavailable.

Expression-scale chronology. The September 10 registration (Row 1 of Table T.1) fixed the metric, splits, comparator family, A1A_1 member, coverage rule, δ\delta, ρ\rho, α\alpha and R=10R=10. Table B.1 records the fixed training recipes and input scales. Row 2 discloses earlier test access from three one-epoch pipeline checks and model-free comparator scores, excluded from campaign counts.

Row 3, dated September 12 after seventeen completed campaign runs, concerns only the scoring of existing predictions. GEARS had been trained on raw counts and STATE and CPA on log⁡(1+x)\log(1+x) inputs, whereas the analysis loader read raw-count truth for all three because Row 1 had omitted the scoring dataset version. The amendment aligned truth, control subtraction and comparator fitting with each model's input scale. Existing predictions and training hyperparameters were retained, and no completed seed was rerun. The recorded criterion was agreement with independent wrapper scores, not score maximization.

Those seventeen runs were GEARS seed 1, STATE seeds 1 to 3 and CPA seeds 1 to 3 on K562, and GEARS seed 1, STATE seed 1 and CPA seeds 1 to 8 on RPE1. Their saved arrays are finite, and all sixty retained configurations match this scale assignment. These files corroborate the record without proving its historical timing. Across the four raw-or-log scoring combinations over the two screens, the K=0K=0 tail 0.2510=9.54×10−70.25^{10}=9.54\times10^{-7} lies below α/4\alpha/4. Corollary 1 covers fixed scoring choices, not adaptive training recipes. Appendix A and Table T.1 now separate these two timelines.

Coverage and membership. Both arms of Table P.1 score all 404 K562 and 441 RPE1 test targets (Appendix P). Covered means that a prediction exists, including for zero-feature targets. The input tensors carry nonzero training-derived features for 371 and 386 targets, the exact held-out genes present in each expression panel. The remaining 33 and 55 targets receive the control's zero-vector fallback and are listed in the companion comment below.

Nonzero-feature verdicts. Let NN be the full target count, mm the zero-feature count, WallW_{\mathrm{all}} the full-target mean excess and cˉ0\bar c_0 the comparator's mean score on zero-feature targets. Since each model correlation is at least −1-1,

Wnonzero≤NWall+m(1+cˉ0)N−m.W_{\mathrm{nonzero}}\le\frac{N W_{\mathrm{all}}+m(1+\bar c_0)}{N-m}.

Retained per-target comparator scores, selection maps and linear-predictor scores determine cˉ0\bar c_0 for all six comparators and reproduce their stored full-target means. The largest upper bound over runs and comparators is −0.0383-0.0383 on K562 and +0.0056+0.0056 on RPE1, both below δ=0.0139\delta=0.0139. Thus K=0/10K=0/10 against all six comparators on both nonzero-feature subsets, with no imputed model scores. Exact restricted means are unavailable because predictions survive for only six of the twenty runs and per-target model scores were not retained.

Release and training evidence. The fixed, untuned recipe uses arc-state 0.10.5/cell-load 0.10.3, log⁡(1+x)\log(1+x) raw counts on the full panel without library-size normalization, training-only ten-PC features, state_sm, 4,000 steps, batch size 2, cell sets of 32 and validation every 20 steps. In six surviving traces, runtime perturbation mappings match the input feature tensors. Each records 200 validation observations, finite losses, a 4,000-step terminal message and prediction from final.ckpt, without test-score checkpoint selection. Retry traces cannot be reverified for all twenty runs. Late validation loss still improves, so convergence and optimal release performance are not shown. The post-hoc 40,000-step follow-up uses the same recipe and five seeds per screen (Appendix P, Table P.2). Rescoring the six surviving 4,000-step prediction arrays reproduced their stored scores exactly. Minimum validation loss occurs between steps 30,400 and 38,600. All ten scores rise over the same-seed 4,000-step run, by 0.128 on K562 and 0.094 on RPE1 on average. Mean excess over registered A1A_1 is −0.098-0.098 and −0.100-0.100, yet every comparator refutes on both screens (K=0/5K=0/5 in both arms), rejecting success rates ≥0.75\ge0.75. The README's larger model and batch size remain untested.

Contribution. The A3A_3 envelope adds no refutation once a constituent fixed comparator refutes on the same targets and score. Norman changes the comparator family rather than isolating authority. Our contribution is a comparison specification and conditional empirical evidence, not a new statistical principle.

AuthorsOpenReview ↗

Author Response to Reviewer L1u4 (Part 2, Continued)

Selector strength and the margin. We agree that the stronger training-only selector does not improve on training-selected ridge (Table I.2). Its role is to show that ridge is not itself a weak comparator, so GEARS's positive excess over the registered A2A_2 selector does not survive a stronger selector. The margin is calibrated on run-to-run variability across six GEARS development seeds (Appendix D) and does not measure biological utility. Our conclusions remain conditional on mean per-target delta correlation and the tested recipes.

Zero-feature targets (all other frozen test targets carry nonzero features, Appendix P).

K562 (33 of 404). ABCG1, BGLAP, CCDC144A, CDAN1, CENPI, CPAMD8, DNLZ, EIF3CL, ESPN, F8A1, FRG2, HAPLN2, HIST1H2BB, HIST2H3A, KRTAP4-2, LCE1C, LPA, MRPL53, NOMO3, NPB, NRF1, OR4F4, RNF103, SGPP1, SIGLEC14, SLC16A5, TREX2, TULP1, ZNF133, ZNF236, ZNF253, ZNF674, ZNF718.

RPE1 (55 of 441). CA5A, CCDC144A, DDN, DEPDC5, DMRTA2, DPY19L2, EPS8L1, FAM102B, FAM229A, FOLR3, FOXL2, GAB2, HCRTR1, HIST1H2BN, HYPK, KRTAP4-2, LCE1E, LCN10, MED18, MED9, MEIS3, MEMO1, NBPF15, NFKBIE, NOMO1, NOP9, NPB, OR2T5, OR4F4, OR7A17, OTX1, PLA2G10, PTCD1, RGPD6, RNF123, RPTOR, S100A1, SIGLEC14, SLC16A5, SPDYE2, STAT5A, STAT5B, TBX1, TDGF1, TRAIP, TREML2, TUT1, WDR24, WDR7, YPEL1, ZBTB14, ZFP69B, ZNF133, ZNF284, ZNF492.

Reviewer_L1u4OpenReview ↗

Reviewer response

Thank you for your explanations. I'll keep my initial score (3).

Thank you for taking the time to read our response and for your constructive feedback.

Reviewer_ArfZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a framework for evaluating perturbation predictors against baseline comparators, outputing one of 3 verdicts the superiority of a predictor: supported, refuted, or inconclusive.

Baseline comparators that a perturbation predictor is compared against could be a seemingly-trivial baseline, like a predictor that for each unseen perturbation simply predicts the average expression shift in the training set. Choosing among comparators to benchmark a predictor is not trivial. The comparators are stratified by their "authority", defined as the information and rules used to select the baseline. Comparators with 3 types of authority (A1, A2, and A3) are used. A verdict is concluded based on the comparators and statistical tests.

Critical strengths
  • Evaluating perturbation predictors is a tricky task; different benchmarks/metrics produce different ranking of methods, and metrics are prone to inflate when, e.g., the perturbation effect is small. Therefore, the framework of the paper seeks an invaluable goal.
  • The framework is extensively applied to 3 state-of-the-art perturbation predictors (CPA, STATE, GEARS).
  • It shows that the 3 methods cannot easily get a success verdict, showing the effectiveness of the proposed evaluation framewor,.
Critical weaknesses
  • Is the method available as a tool, readily available to perturbation predictor models?
  • Two quesions/concerns regarding Eq. 3
    • The criterion used is the correlation between the predicted and ground-truth shift in gene expression. But perturbations tend to induce minor changes in expression, thus correlation is heavily inflated by the genes which are not supposed to change?
    • To arrive at s(g,e)s(g,e), the correlation is averaged over perturbations. Could other choices be beneficial?
  • In Eq. 4, minimum value across environments is calculated. Is it the only choice? could other choices be explored or discussed?
Questions for authors

Please see my notes above.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer ArfZ

Thank you for these questions. The anonymous supplement provides offline decision-layer code for supplied score tables rather than an end-to-end predictor runner. Across the evaluation choices you raise, the comparisons against A1A_1, training-selected ridge and A3A_3 remain negative under rank correlations and median, trimmed-mean and alternative environment aggregations, whereas GEARS against the weak A2A_2 selector is sensitive to them.

Tool availability. The anonymous supplement supports offline reliability analysis of supplied scores. It includes the decision-layer code, unit tests, retained run summaries and per-target scores, with commands to reproduce the reported tables. Applying it to a new predictor still requires upstream integration, as raw expression matrices, trained checkpoints, full prediction profiles, and parts of data loading and comparator fitting are not bundled. It is not a standalone one-command evaluator for new models.

Equation 3 and weak effects. Subtracting the control mean avoids directly rewarding agreement on baseline expression across non-changing genes, which otherwise dominates flat expression correlations (Appendix D). However, full-panel delta correlation uses every panel gene (8,563 in K562 and 8,749 in RPE1, Table C.1) rather than a top differentially expressed subset, so weakly responding genes still enter the score. Control subtraction alone does not eliminate bias from shared control estimation, weak perturbation shifts or responses common across perturbations, and subtracting a comparator score does not guarantee exact bias cancellation. Rather than filtering genes, we evaluate comparators under the identical full-panel score so that the excess measures improvement beyond signals captured by simple baselines.

The comparator family spans training-average shifts and baseline-expression and cell-cycle shortcuts. To test sensitivity to the control reference, Appendix K splits control cells into disjoint, batch-stratified halves and retains 178 of 180 run-level refutations. The two exceptions are the covered and control-filled arms of one partition for GEARS on K562 against registered A1A_1 (Table K.1), which become inconclusive while training-selected ridge still refutes. Replacing Pearson with Spearman rank correlation in Appendix M (Table M.2) also keeps mean excess negative against A1A_1, ridge and A3A_3 throughout. These checks support the comparative verdicts while leaving the biological limitations of delta correlation in place.

Averaging over targets. Alternative target summaries address different estimands. Equal-weight averaging treats every held-out perturbation equally, a median reflects typical target performance, a lower quantile emphasizes poor-tail targets, and effect-size weighting emphasizes strong perturbations. Concatenating all targets into one pooled correlation alters both weighting and geometry and increases seed standard deviation 24-fold on our development predictions (Appendix D, Table D.1), though that observation on one split does not prove a universal advantage for averaging.

New Appendix X and Table X.1 summarize the retained targetwise model-minus-comparator differences on the primary covered arm using the median and the 10%-trimmed mean. Against A1A_1, training-selected ridge and A3A_3, every run stays negative in all six model-screen cells under both summaries. GEARS against A2A_2 is sensitive to the summary. On RPE1 its median excess is negative in all ten runs even though its registered mean excess is positive. We report these summaries descriptively without recalibrating δ\delta or issuing new confirmatory verdicts.

The minimum in Equation 4. Taking the minimum across environments tests whether a run succeeds on every declared screen. An across-screen mean tests average gain and can mask a loss on one screen, whereas a maximum tests gain on at least one screen. We therefore report single-screen verdicts alongside the joint minimum. Against A1A_1, ridge and A3A_3, every run falls below the comparator on both screens, so every aggregation between the minimum and the maximum yields K=0/10K=0/10 (Appendix X). By contrast, for GEARS against A2A_2, the seed-paired count rises from 6/10 under the minimum to 10/10 under the cross-screen mean or maximum. Appendix L also shows that the registered joint counts are invariant to seed pairing in these data. In general, the cross-environment aggregation should match the substantive claim and be fixed before confirmatory testing.

Reviewer_ArfZOpenReview ↗

Reviewer response

Thank you for your explanations. I'll keep my initial score (3).

Thank you for taking the time to read our response and for your constructive feedback.

Reviewer_cmdrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces a framework to compare virtual cell predictors with simple baselines in a more strict way, to counter statements such as “model X can/cannot beat a simple baselines”. They introduce six different baselines that are blind to the perturbation, and three “authorities” over two different cell lines. Their results are interesting: None of the models can beat a baseline that is blind to the gene being knocked out. While this result is not new, the authors make a more careful and detailed analysis than those presented in the past

Critical strengths

Presentation

The paper is generally well written, aside from a very dense notation (see weaknesses). The figures are clear, and the writing is good. I particularly liked figure 1. The results are nicely laid out

Limitations

The authors are honest when it comes to acknowledging the limitations of their own work, such as the limited number of cell types. They also acknowledge that most of their main results have been claimed before, and that honesty is appreciated.

Statistical rigor

The paper is very rigorous when it comes to the statistical analysis. The authors are correct that, particularly in the area of virtual cells, the term “simple baseline” is broadly used to mean different things, so it is useful that they make these definitions. Their results also follow careful statistics. The only issue I have with this is the choice of threshold and margin (see weaknesses)

Main result

While not new, their main result (these models cannot beat baselines) remains very interesting, especially after a more detailed analysis

Critical weaknesses

Novelty

While I enjoyed reading the paper, and found it well laid out, and generally useful, it did leave me wondering if it is a good fit for the conference. The framework is useful, and can be cited and reused in future virtual cell analysis, so it does have value. However, from a perspective of the novelty, the main contributions is a framework to report how the baseline was chosen and to test each training run with a binomial test, and applying it to three models to reach some conclusions that had largely already been established. This makes me wonder if a more specialized journal would be a better fit.

Notation

I find the notation at times unnecessarily convoluted. Introducing these terms for predictors, comparators, environments (which if I understand correctly, are just different cell types). At times while reading it felt like this overly convoluted notation was hiding the relatively simple stuff that was going on under the hood

Limited data and models

The analysis is limited to only two cell types, which limits the potential impact of the contribution. Similarly, there are many models aside from the ones benchmarked in this paper (TxPert, DeSCOPE, scGPT and many more) including some that claim to beat basic baselines. In fact, only GEARS is used in its intended regime, per the authors’ admission.

Choice of threshold

The authors use Bouthillier's 75% threshold (which as an aside I personally do not like, as it seems quite arbitrary) but as the authors themselves acknowledge, with this threshold, the outcome can never be “supported”. While this is softened by the actual results (mostly 0 wins) this remains an issue.

Choice of margin

The choice of margin δ\delta is not very principled and poorly justified (standard deviation of GEARS in one split), even though the conclusions are not particularly affected by it

Questions for authors

Could you apply this framework to more models such as TxPert, DeSCOPE, scGPT

What is the point of A2A_2? Is it just to show a weaker baseline?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer cmdr

Thank you for the review. The campaign and reference-preprocessed GEARS refutations against the fixed and training-selected comparators hold at any δ≥0\delta\ge0 and at reliability thresholds well below 0.75. Every run in those comparisons scores below its comparator.

Margin. δ=0.0139\delta=0.0139 is GEARS's seed standard deviation on a K562 development split disjoint from the confirmatory test targets (Appendix D). It reflects run variability, not biological utility. The best of the sixty campaign runs against A1A_1 has excess −0.000243-0.000243. With reference preprocessing across twelve fresh splits per screen, GEARS's best excess against the training-selected comparator is −0.069-0.069 on K562 and −0.058-0.058 on RPE1 (Table O.1). These comparisons have zero successes at every δ≥0\delta\ge0 (new Appendix X). GEARS against the weak A2A_2 selector is the exception and remains sensitive to the margin and the score.

Threshold. Support under ρ=0.75\rho=0.75 is unattainable with the campaign's ten-run budget, not in general. At ρ=0.75\rho=0.75 and individual α=0.0125\alpha=0.0125, ten runs refute at K≤3K\le3 while 10/10 stays inconclusive, separating inconclusiveness from refutation. Support becomes attainable at 16 runs. The thirty-run Norman control reaches 30/30 support against its registered fixed comparator and envelope (Appendices A and R), showing attainability for that control's run generator, not power for arbitrary neural pipelines. For a fixed specification with K=0K=0, exact binomial refutation needs only ρ≥1−α1/R\rho\ge1-\alpha^{1/R}, about 0.355 at R=10R=10, 0.306 at R=12R=12 and 0.136 at R=30R=30 (Appendix X).

Novelty and fit. The binomial test and fixed, selector and oracle comparisons are standard, and we do not present them as new. For machine learning benchmarks, the paper contributes a reusable comparator specification for training-run evaluations and stronger conditional evidence under it. Domain generalization benchmarks report what data model selection may use (Gulrajani and Lopez-Paz, 2021), and we apply that discipline to the comparator. Comparator choice changes verdicts. In the registered linear-predictor test on K562 raw counts, the same thirty bootstrap refits are supported against A2A_2 (30/30) and refuted against registered A1A_1 (0/30, Appendix Q). Under ρ=0.75\rho=0.75, any ten-seed benchmark can refute such a claim and cannot support it.

Models and data. A follow-up comment and new Appendix Y test DeSCOPE and full scGPT post hoc, since the earlier scGPT-embedding ridge head did not test full scGPT. We did not run TxPert. Our tests cover three screens across two cell lines (K562 and RPE1, since Adamson is an independent K562 screen, Section 7) and concern only the tested recipes, scores and screens. Initial CPA and one-hot STATE lie outside their supported regimes and cannot ground a general negative result, which the GEARS and STATE follow-ups address. Against the primary training-selected comparator, reference-preprocessed GEARS gives 0/12 successes on each Replogle screen, and corrected-label Adamson gives 0/30 on one split and across thirty fresh splits (Appendices N and O). Adding training-derived features to STATE's untuned 4,000-step recipe gives 0/10 on both screens, and nonzero-feature bounds preserve its refutations at δ=0.0139\delta=0.0139. A post-hoc 40,000-step test (five seeds per screen) raised STATE's score in all ten runs, by 0.128 on K562 and 0.094 on RPE1 on average, yet every comparator still refutes it on both screens with K=0/5K=0/5 (revised Appendix P, Table P.2).

The role of A2. A1A_1 and A2A_2 both exclude test responses. A2A_2 selects a family member for each target using training-control gene features and cross-fitted training labels, without held-out responses. Intended to improve on A1A_1, it scored lower on the development split and was kept as registered (Appendix E). Its failure concerns this selector, not selection in general, and shows why authorities are unordered (Appendix A). GEARS's positive mean excess over A2A_2 vanishes against training-selected ridge and nested selection (Appendix I). Our principal negative results rest on the stronger comparators, while A2A_2 shows why superiority claims must report comparator strength.

Notation. The procedure scores each target's predicted expression change, averages over targets, subtracts the declared comparator, counts complete training runs clearing δ\delta, and applies a binomial test. An environment is a Perturb-seq screen (K562 and RPE1 in the main campaign), and the cross-screen minimum asks whether a run succeeds on every declared screen. Section 4 now opens with this plain summary before the formal notation.

AuthorsOpenReview ↗

Additional Benchmark Results (DeSCOPE and scGPT) for Reviewer cmdr

Thank you again for asking us to apply the framework to additional models. During the discussion period we evaluated DeSCOPE and full scGPT under the same specification (new Appendix Y, Table Y.1). Both are post-hoc tests outside the registered campaign. Each runs the model's own tutorial recipe without tuning, and the reading rule for each outcome was fixed before any score was computed. Both reuse fresh splits from the follow-up tests with the same prepared training data, comparators and scorer, pairing every run with the GEARS run on the same split. At twelve runs per screen, a comparator refutes at K≤4K\le4 and primary support is unattainable.

DeSCOPE on Replogle K562 and RPE1. DeSCOPE's documentation reports strong performance on unseen genes, the regime of these splits. We trained its standard model on the 24 fresh splits of the GEARS multi-split test (Table O.1), twelve per screen, on the exact 80,000 training cells used for GEARS and the comparators. On K562, the primary training-selected comparator does not refute DeSCOPE. DeSCOPE exceeds that comparator in all twelve runs (mean +0.016+0.016, range +0.006+0.006 to +0.025+0.025) and clears δ=0.0139\delta=0.0139 in nine, which is inconclusive at ρ=0.75\rho=0.75. When the 5 to 18 of 404 K562 test targets without an ESM2 gene embedding are filled with controls, only three runs clear δ\delta, a refutation in that arm, so the two coverage arms disagree and the verdict is inconclusive (registered A1A_1, A2A_2 and nested selection are likewise inconclusive on K562). Both the per-target envelope A3A_3 and the linear gene-embedding baseline refute DeSCOPE on K562 (K=0/12K=0/12 in both arms). On RPE1, 5 to 16 of 441 test targets lack embeddings, and every comparator refutes DeSCOPE (K=0/12K=0/12 in both arms, mean −0.037-0.037 against the training-selected comparator). Scored across all test targets with missing predictions control-filled for both models, DeSCOPE outscores GEARS in all 24 paired runs, by a median of +0.095+0.095 on K562 and +0.061+0.061 on RPE1.

Full scGPT on Adamson. Because the submission's scGPT-embedding ridge head does not test the full model, we fine-tuned scGPT end to end from its public whole-human checkpoint using its perturbation tutorial configuration (15 epochs, learning rate 10−410^{-4} decayed by 0.9 per epoch, batch size 64, keeping the epoch with the highest validation Pearson correlation). We ran the first twelve fresh Adamson splits of Table N.2, using the paired GEARS run's validation targets for model selection. Three runs ran out of GPU memory in epoch 2 and were repeated with the same seed, accumulating weighted gradients across two half-batches before each optimizer step at nominal batch size 64. Every comparator refutes full scGPT (K=0/12K=0/12 in both arms, which coincide because all 16 test targets are predicted). Its excess over the training-selected comparator averages −0.101-0.101 (range −0.189-0.189 to −0.020-0.020), so every run falls below the baseline, and the verdicts are unchanged at δ=0.0346\delta=0.0346 and ρ=0.5\rho=0.5. Paired with GEARS on all 16 test targets (with GEARS's unpredicted targets control-filled), scGPT scores higher in 9 of 12 runs (median +0.038+0.038, range −0.041-0.041 to +0.112+0.112).

These tests leave the main negative findings intact while illustrating a distinct outcome under the specification. Full scGPT under its tutorial recipe is refuted on the independent screen alongside GEARS, despite outscoring GEARS on most splits. DeSCOPE provides a case that escapes refutation by the training-selected comparator on one screen, showing how the specification keeps a consistent positive excess that twelve runs can neither confirm nor reject distinct from support. Both tests evaluate one untuned tutorial recipe per model, and we did not run TxPert during the discussion period.

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.