acceptodds
Under review as a conference paper at ICLR 2027

The Construct Is Part of the Target: Rethinking Perturbation-Model Evaluation

Abstract

Reagent-level variation is biologically established, but perturbation-model benchmarks typically evaluate predictions at the gene level. We ask whether model evaluation changes depending on which reagent is used to perturb a gene and whether that reagent is available in source data. All 2,053 genes observed in all four core CRISPRi panels reuse at least one complete construct across all four contexts, so cellular context holdout alone does not imply construct holdout. Alternative constructs produce recurring response differences beyond matched cell resampling; external CRISPR KO and CRISPRi libraries also show differences beyond same-guide resampling. Holding predictions fixed, changing the realized target changes absolute model scores; released State predictions also show sensitivity to target realization beyond same-construct resampling. Under global construct exclusion and matched cell counts, all 16 comparisons show higher mean Pearson when the target construct is present in source data than when it is withheld. Construct transfer also changes the relative MAE performance of the two fitted predictors, while same-construct resampling does not reproduce the shift. Their mean Pearson ordering also reverses descriptively in all four contexts. A State architecture trained from scratch under the same exclusion design shows material Pearson degradation in RPE1 and Jurkat. Perturbation-model evaluation therefore requires specifying both target realization and which constructs are present in source data.

Primary area:
datasets and benchmarks
Keywords:
single-cell perturbation modeling, perturbation response prediction, gene perturbation, benchmark evaluation, computational biology, distribution shift

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tUgJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how CRISPR construct identity affects genetic perturbation model evaluation. Across multiple datasets, the authors show that different constructs targeting the same gene produce distinct responses and can change model evaluation scores. Under global construct exclusion, the authors further show differential transfer penalties across models and descriptive ranking reversals. The paper argues for benchmarking that is construct-aware and explicitly specifies target realization and construct availability in training data.

Critical strengths

1. An underexplored aspect of evaluating perturbation prediction

The authors show that the gene which is perturbed does not uniquely determine the experimental response. In particular, how construct identity overlaps with training and evaluation can affect scores and sometimes model rankings.

2: Good experimental design

The paper shows how construct identity affects scoring across multiple external CRISPR dataset and cell lines, using careful controls (e.g., matched cell counts, same construct resampling controls, etc.).

Critical weaknesses

The generality of the demonstrated effects on model comparisons is limited The paper convincingly shows how construct identity can alter absolute performance, however, the evidence for broader implications for model comparisons is less established. The main performance analysis compares ridge regression with an adjusted fMLP on relatively small cohorts. The matched-cell count MAE gap changes are statistically supported across contexts, however, only K562 exceeds the prespecified materiality threshold. The State experiments do show sensitivity in absolute performance, but do not show alterations in rankings among modern perturbation models.

Questions for authors

1. Clarification: Significance of changes in model rankings

Could the authors clarify how the findings support adjustments in pratical model selection, rather than differences in transfer penalties? Is there evidence that the ranking reversals you observe are robust under paired uncertainty?

2. Additional analysis: Performance comparisons for State

Given that the from-scratch trained State and predictors are evaluated under the same global construct exclusion framework, could the authors compare State directly against ridge regression and/or fMLP on the shared evaluation targets, with paired uncertainty? Such a comparison would help establish whether the construct realization can materially change conclusions about model selection beyond the primary fMLP ridge comparison.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer tUgJ

Thank you for the careful review and for recognising the controls, which remain in the revision (§3.1, Supp. S10, Table S13). Your central point is whether the construct changes model comparison, beyond absolute scores. The revision's main addition answers it: six models, each fitted under standard and under construct-aware evaluation, on the same 185 gene–cell-type pairs (§5).

Terms, as in the revised PDFs. Standard evaluation keeps every construct in training and scores the gene-pooled response. Construct-aware evaluation withholds the scored construct from training and scores that construct's own response; the gene's partner construct stays in training. CA is centroid accuracy, the primary score (§2). Points run from 0, the no-change baseline, to 100, a technical duplicate. Table A59 gives each claim's timing (predeclared or post hoc) and its error control.

W1. "the evidence for broader implications for model comparisons is less established"

Under CA the ranking changes, and so does the top model.

  • The plan's ranking analysis on the 185 pairs returns "THE RANKING CHANGES" between standard and construct-aware evaluation (Kendall τ = −0.429; plan analysis S1, Supp. S25).
  • Withholding the scored construct changes the top model under CA from Perturb Mean to ridge (§5.1; App. M, Table A36). The intervals and the scope are in Q1b.
  • Perturb Mean's standard advantage collapses under both scores. On the 185 pairs its cosine CA is 0.7892 under standard evaluation and 0.5020 under construct-aware evaluation, where chance is 0.5 (Table A7).

W2. "compares ridge regression with an adjusted fMLP"

The main comparison now covers six models: State trained from scratch, CPA, biolord, ridge, Perturb Mean and Linear-PCA, with the fMLP as a reference (§2, §5). The fMLP–ridge comparison moved to App. G (CA) and Supp. S10 (the submission's MAE analysis, unchanged). Under CA the fMLP loses more than ridge from the partner construct's target to the scored construct's. At fixed hyperparameters, expanded benchmark (399 pairs), the gap change D is 0.0237 [0.0158, 0.0325], significantly positive in all four cell types (App. G, Tables A22–A23; this CA readout was designated after the MAE results were known).

W3. "on relatively small cohorts"

The revision enlarges the benchmark and reports what it can detect. The comparison exists for multi-construct genes, which are 5.6–5.7% of genes and 9.8–14.1% of cells per screen (§1).

  • The submission had 197 pairs (67 genes); the expanded benchmark has 399 pairs (113 genes). The six-model benchmark uses the 185 original pairs (62 genes) whose gene has no third construct (§2, Table A2).
  • With 113 genes, construct-aware comparisons detect model differences of 0.006–0.011 CA at 80% power; the benchmark separates models 4.4–7.9 points apart (App. N).
  • On the 185 pairs, every construct-aware CA margin is bounded at 7.4 points or less (Table A8).

W4. "only K562 exceeds the prespecified materiality threshold"

Three printed results clear the threshold beyond K562. The sentence you cite is the per-cell-type MAE form of the fMLP–ridge gap change D, kept unchanged in Supp. S10.

  • Pooled over cell types, D is 0.0113 [0.0068, 0.0158], above the 0.005 threshold (Table S20).
  • On the expanded benchmark, D clears the threshold in K562 and HepG2 (Supp. S12).
  • For State against ridge, the MAE gap change is +0.016 to +0.023 and clears the threshold in all four cell types (Supp. S12, Table S28).

W5. "do not show alterations in rankings among modern perturbation models"

Under CA, State's order against ridge reverses. The plan's ranking analysis lists State vs ridge as a robust reversal: State leads ridge under standard evaluation, and ridge leads State once the scored construct is withheld (Supp. S25, "Ranking", supp. p. 63). The pooled scores and the paired evidence are in Q2a.

  • In the submission's MAE and Pearson design, the best-scoring model changes between the partner construct's target and the scored construct's in 7 of 8 cell-type-by-metric cells (post hoc; Supp. S12, Table S29); the predeclared test for a robust reversal involving State is not met there. MAE and Pearson do not order known skill (§4.2), so the ranking claim rests on CA.
  • Among modern perturbation models (here State, biolord and CPA), the set that beats the no-change baseline changes. Standard evaluation puts State and biolord significantly above it, at 33.8 [18.2, 48.7] and 22.9 [6.1, 37.4] points; CPA is not (CA; Supp. S25).
  • Five of six models significantly outperform the no-change baseline in standard evaluation, but none retains a significant advantage under construct-aware evaluation (CA; 185 pairs; §5.1, Fig. 1C). State is then significantly below the baseline, at −5.2 [−13.8, −0.1] points (CA; Supp. S25).

Q1a. "how the findings support adjustments in practical model selection, rather than differences in transfer penalties?"

A difference in penalties matters for selection when it reverses an order, and here it does. Under CA, standard evaluation selects Perturb Mean and construct-aware evaluation selects ridge (§5.1; App. M).

  • Training alone reverses the order on one fixed target. Scored on the same construct target, ridge − Perturb Mean is −0.027 [−0.037, −0.018] with the scored construct in training and +0.008 [0.001, 0.015] without it (CA; Table A15; expanded benchmark, conventionally tuned, full cell count; error control in Q1b). On the 185 pairs, the same fixed-target comparison lists Perturb Mean vs ridge as a robust reversal (plan analysis S1b, Supp. S25).
  • Tuning selects differently. Conventional and construct-aware tuning select different configurations in all four cell types (Supp. S8).
  • Standard evaluation credits two gains that a new construct does not supply. Target change comes from scoring the gene-pooled response instead of the construct's own. Leakage comes from having the scored construct in training. Under CA they are worth 26.0 and 13.0 points to State (Fig. 3A, Table A9; 185 pairs). State's target change is significant under both scores. Its leakage is significant under CA; under cosine CA its interval includes zero, at +0.0391 [−0.0018, 0.0827] (three seeds; Table A12).
  • The gains differ by model. Leakage is 14.1 CA points for Perturb Mean and 0.7 for ridge, significant for both under CA and cosine CA; target change is 30.6 points for Perturb Mean (Tables A5, A9).
  • Both gains move the ridge–Perturb Mean gap. At full cell count on the expanded benchmark, the plan's verdict under CA is that target change carries the swing; leakage is 0.31 [0.24, 0.38] of it (App. D). Under cosine CA the label is that both contribute (Table A21).
  • The loss differs by model. Where the scored construct responds, withholding it removes 85% of Perturb Mean's margin and 8% of ridge's (§5.3, App. H; expanded benchmark, 177 pairs, full cell count).

In practice: report both standard and construct-aware margins, and select on construct-aware evaluation when the model will be used on new constructs. §6 states that the training split should match the generalization task (steps 2 and 4).

Q1b. "Is there evidence that the ranking reversals you observe are robust under paired uncertainty?"

Yes. A flip requires a pair's gap to be significant under both evaluations, with opposite signs (App. M). Ridge flips against Perturb Mean, Linear-PCA and the fMLP, at half and at full cell count. Each interval in Table A36 is on the difference between two models over the same gene–cell-type pairs, not on each model separately (CA; expanded benchmark, 399 pairs; conventionally tuned; 95% gene-cluster bootstrap). At full cell count:

PairStandardConstruct-aware
ridge − Perturb Mean−0.105 [−0.130, −0.082]+0.008 [0.001, 0.015]
ridge − Linear-PCA−0.033 [−0.051, −0.018]+0.012 [0.007, 0.018]
ridge − fMLP−0.020 [−0.035, −0.007]+0.009 [0.006, 0.013]
  • The ranking-change test, at least one flip, has p = 6.0×10⁻⁵ and is retained by Holm (Supp. S35; App. X).
  • Ridge's lead in the first row, +0.008 [0.001, 0.015] (conventionally tuned, full cell count; retained by FDR control, not Holm), is a CA result.
  • At full cell count, Perturb Mean's standard lead and ridge's construct-aware lead are both significant under conventional tuning and with fixed hyperparameters; under the specificity floor and under construct-aware tuning, the ridge − Perturb Mean flip weakens there (Table A37). At half cell count every configuration repeats the lead change (App. M).
  • The fMLP–ridge flips are 2.2–11.8× the seed standard deviation, with the same sign in 3 of 3 seeds (Supp. S35).
  • The six-model benchmark agrees under CA. On the 185 pairs, ridge exceeds Perturb Mean under construct-aware evaluation by +0.0102 [0.0021, 0.0195] CA; under cosine CA the difference is +0.0175 [−0.0379, 0.0743] (Supp. S25).
  • The reversal in the submission was a descriptive mean-Pearson ordering (Supp. S10). Pearson fails every applicable known-skill ladder (§4.2), so the revision tests reversals under CA, where the same fMLP–ridge pair flips (third row).
  • Perturb Mean's advantage collapses under both scores. The swing is 0.79 [0.71, 0.90] of its standard margin under CA (App. M) and 0.879 [0.729, 1.044] under cosine CA (Table A21).

Q2a. "compare State directly against ridge regression and/or fMLP on the shared evaluation targets, with paired uncertainty"

On shared targets, yes. State, ridge and the fMLP are fitted under both evaluations and scored on the same 185 pairs and the same targets (Table A7). The paired evidence the revision prints is listed below. Under CA it pairs each model with the no-change baseline and with its own fit that saw the scored construct; in MAE it pairs State with ridge and with the fMLP.

ModelPooled CA, standard → construct-awareConstruct-aware CA margin [90%]Leakage under CA, points [95%]
State0.5597 → 0.4831−0.0083 [−0.0151, −0.0015]13.0 [4.0, 22.5]
Ridge0.5185 → 0.4972+0.0058 [−0.0014, 0.0138]0.7 [0.5, 1.0]
fMLP0.5214 → 0.4884−0.0030 [−0.0075, 0.0014]2.0 [1.1, 3.2]
No-change baseline0.4935 → 0.4914——

By pooled score, State leads ridge under standard evaluation and ridge leads State once the scored construct is withheld; the plan's ranking analysis lists the pair as a robust reversal (Supp. S25, "Ranking"). The pooled scores are from Table A7, whose intervals are per model, not paired differences. The margins are from Table A8 (post hoc; 90% for its two one-sided tests): each is a model minus the no-change baseline over the same pairs, on the shared construct target. Leakage is from Table A9 (post hoc): each model's fit that saw the scored construct minus its fit that did not, on that target.

  • Paired CA margins in points, standard evaluation (95% intervals; Supp. S25). State has 33.8 [18.2, 48.7] and ridge 12.7 [3.2, 20.8].
  • The same margins with the scored construct withheld. Ridge's interval includes zero, at 3.7 [−2.3, 8.1]; State is significantly below the no-change baseline, at −5.2 [−13.8, −0.1].
  • Paired leakage under CA (last column). State's and ridge's stay significantly positive with Holm-adjusted intervals (Table A5). State's is significantly positive in each of three training seeds (Table 1; cosine CA: Q1a).
  • Paired MAE gap changes, State vs ridge (the submission's design; Table S28, one State seed). State loses more than ridge from the partner construct's target to the scored construct's in every cell type. The gap change is +0.016 to +0.023, above the 0.005 threshold in each, and 0.0188 [0.0126, 0.0251] pooled (Table S20). By point estimate, State has the lower MAE on the partner construct's target and ridge on the scored construct's, in all four cell types (Supp. S12).
  • Paired MAE gap changes, State vs fMLP (Table S28). The gap change is positive in all four cell types by point estimate and significant in RPE1, at +0.0101 [0.0033, 0.0184]. Both MAE comparisons are retained by Holm (claims A27 and A28, Table A61).
  • Paired add-back contrast (Table A57). Returning the scored construct to training lowers State's MAE on the construct target more than ridge's, by +0.019 [0.006, 0.042] in RPE1 and +0.020 [0.007, 0.036] in Jurkat.

Q2b. "whether the construct realization can materially change conclusions about model selection beyond the primary fMLP ridge comparison"

Yes. Beyond fMLP–ridge, the conclusion changes in each case below.

  • Top model. Under CA it changes from Perturb Mean to ridge (Q1b).
  • State and ridge. Their order reverses under CA (Q2a).
  • State's released split, the released model's multi-construct subset (88 pairs, 42 genes). Whether the released model beats a predictor that learns nothing depends on which construct that predictor recalls. State minus the partner construct's training mean is +0.024 [0.007, 0.042] CA, and State minus the scored construct's training mean is −0.061 [−0.081, −0.043] (Table A33).
  • The same split, single-construct genes. There the construct's training mean is the gene's training mean. Pooled over cell types it has CA 0.672 against State's 0.608; State minus this mean is −0.064 [−0.070, −0.058] (§5.2; Supp. S33). It leads significantly in K562, Jurkat and HepG2; State leads in RPE1 (App. K).

On the expanded benchmark (399 pairs, 113 genes; the four models of Q1b), ridge alone has a significantly positive construct-aware CA margin, at +0.0057 [0.0015, 0.0102] (fixed hyperparameters, half cell count; Table A39).

Presentation. The main text was rewritten for readers outside biology, and Table A1 maps each claim to its evidence (general response, Part 2, "What changed").

Resources. The split files, the construct-reuse audit script and the code are available at the anonymous repository https://anonymous.4open.science/r/construct-aware-eval-4B44/.

Thank you again. Your question was whether the construct changes model selection. With six models on shared targets, under CA it changes the ranking, the top model, and State's position against ridge.

I thank the authors for their detailed experiments, which strengthen the paper. Nevertheless, the findings remain limited to relatively small set of multi-construct genes. Additionally, I share reviewer B4kT's concerns with respect to sensitivity to the choice of metric, as well as the substantial changes that have been introduced with regards to core aspects of the paper during the rebuttal period.

In summary, I still find the contribution worthy of acceptance, and maintain my score of weak accept.

Reviewer_B4kTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether perturbation-model evaluation depends on the experimental construct used to perturb a gene, even when the gene label and prediction remain fixed. It distinguishes which construct supplies the scored response from whether that construct is represented in source data. The core evidence comes from four CRISPRi contexts, supplemented by external single-guide CRISPR knockout and interference screens.

The authors show that random cell splits preserving a construct mixture yield higher response concordance than splits separating alternative constructs, and that construct-associated differences recur across experiments and contexts. They then show that changing target aggregation or construct identity changes scores for fixed predictions, including released State predictions. Under global exclusion of one construct while retaining the same gene through another, matched-cell-count evaluation changes the paired MAE gap between fMLP and ridge in all four contexts; Pearson ordering also reverses descriptively. A State architecture trained from scratch shows represented-versus-withheld Pearson differences exceeding the authors' declared threshold in RPE1 and Jurkat. The proposed implication is that benchmarks should specify target constructs, pooling and scoring rules, and source-construct availability in addition to the perturbed gene and cellular context.

Critical strengths

The paper identifies a consequential ambiguity in benchmark design. A cellular-context holdout need not exclude the complete construct used to realize the evaluation target. By retaining the gene while globally excluding an alternative construct, the paper makes this distinction concrete. This is useful for designing reproducible evaluation protocols and interpreting what a reported generalization result actually tests.

Critical weaknesses
  1. The paper establishes that changing constructs changes evaluation scores, but the implications for predictive quality remain unclear. It relies on control-referenced Pearson and whole-profile MAE without showing that these metrics reliably capture perturbation-specific prediction quality on the evaluated cohorts. This matters given the limitations discussed by Viñas Torné et al. (Systema) and Miller et al. The paper’s own results also depend strongly on the response genes scored: random/construct SB ratios shrink substantially when restricted to selected top-response genes (§3.1; Appendix B). These concerns do not invalidate the score comparisons, but they limit their interpretation as meaningful differences in model performance. Practical importance is also insufficiently justified: all four matched MAE-gap intervals exclude zero, but only K562 exceeds the unexplained 0.005 “materiality” margin. The descriptive Pearson ranking reversals are a separate result. The paper should connect these findings more clearly to absolute model scores and metric validity, or explicitly limit its conclusions to the reported endpoints.

  2. The paper is difficult to follow, and important experimental details remain unclear. The problems extend across the overall structure and the prose. Methods and results are not clearly separated, essential definitions appear after the results that depend on them, and understanding the experiments requires piecing together information across the main text and appendices. Dense jargon and compressed phrases, such as “orientation averaging and denominator gated Pearson comparisons,” make the argument unnecessarily hard to parse. The central paired estimates in Tables A7 and A12 also deserve more direct main-text treatment. These issues call for substantial restructuring and clearer, more concrete language throughout. Beyond presentation, the fitted-model evaluation cohorts contain only 57/60/49/31 genes and reuse genes and constructs across contexts, limiting population-wide interpretation even though the paper acknowledges this restriction. The State extension raises a more substantive reporting concern: Appendix J specifies one seed and 100,000 optimizer steps, but does not provide sufficient training configuration, convergence evidence, or detail on the source-only bridge to assess the fitted pipeline fully. It also uses full-depth targets rather than the matched-depth design supporting the fMLP–ridge result. Consequently, I cannot judge whether this experiment establishes sensitivity of a well-validated State deployment or only of the particular adapted, fixed-budget pipeline. The step count alone is not evidence of either adequate or inadequate training.

Questions for authors
  1. Metric validity and response-gene selection (weakness 1). How do the limitations discussed by Viñas Torné et al. (Systema) and Miller et al. affect the interpretation of Pearson-Δ and whole-profile MAE in your experiments? Given the substantial reduction in random/construct SB ratios on selected top-response genes, what supports interpreting the model-comparison changes as differences in perturbation-specific predictive quality? Please distinguish response-profile concordance from model-scoring results and identify any available metric or response-gene sensitivity analyses of the central paired model comparison. If these have not been assessed, please clarify the intended scope of the conclusions.

  2. Practical significance (weakness 1). What justifies the 0.005 MAE and 0.02 Pearson “materiality” margins? Please summarize the absolute model scores and paired gap changes together, distinguishing statistical evidence for a changed gap, an actual ranking reversal, and practical importance. In particular, how should readers interpret the four positive matched MAE-gap intervals when only K562 clears the declared MAE margin, separately from the descriptive Pearson ranking reversals?

  3. Experimental structure and population scope (weakness 2). Could you provide a concise, self-contained account of the central paired experiment: what each model receives, how represented and withheld targets are formed, what the same-construct control tests, and what Tables A7 and A12 establish? Please define the necessary terminology before presenting results and distinguish methods from findings. Please also explain the restrictions yielding the 57/60/49/31 evaluation genes, the overlap across contexts, and which population the conclusions apply to. This would help assess both the argument and its generality without reconstructing the design across appendices.

  4. State implementation and validation (weakness 2). Please provide the training configuration, optimizer/schedule, batch/sampling scheme, checkpoint rule, and construction of the source-only bridge, together with available convergence or source-only validation evidence for Phase B. What supports the adequacy of the single-seed, 100,000-step fit? Is a matched-cell-count State comparison available to assess the contribution of target-depth differences? Please clarify how these choices limit generalization from this adapted pipeline to State more broadly.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer B4kT

Thank you for the careful review. The two weaknesses you raised, metric validity and presentation, set the direction of the revision.

Terms follow the revised paper; Table A2 defines them. The scored construct is the one whose response is predicted (earlier "withheld"). Its partner construct stays in training (earlier "represented"). Table A59 gives each claim's timing (predeclared or post hoc) and its error control.

Summary

  • Metrics (W1, Q1, Q2). We agree. Pearson-Δ and MAE do not rank predictions of known skill in order (§4.2), so §5 compares models with centroid accuracy (CA) and cosine CA.
  • Tables A7 and A12 (W2). The main text's paired comparison is now a six-model benchmark (§5.1; Fig. 1C, Fig. 3A, Table 1). The fMLP–ridge estimates are in Supp. S10 (MAE, as registered) and App. G (CA).
  • Presentation (W2, Q3). In the rewritten main text, terms and design come before results.
  • State (Q4). The leakage claim, under CA, rests on three training seeds in four cell types (Table 1).

W1, Q1. Scope ("connect … to absolute model scores")

Standard evaluation credits margins that do not carry over to an unseen construct.

  • Margins are in points, a scale fixed by two reference predictions: 0 is the no-change baseline and 100 the technical duplicate, the other half of the target's own cells (§2).
  • Under CA, five of six models significantly outperform the no-change baseline in standard evaluation, but none retains a significant advantage under construct-aware evaluation (185 pairs; §5.1, Fig. 1C).
  • The conclusions concern these margins and how they change. They do not claim good absolute prediction of an unseen construct.

W1, Q1. Do Pearson-Δ and whole-profile MAE capture prediction quality on these cohorts?

No, and model comparison no longer rests on them.

  • The known-skill check. We scored predictions of known skill, with no fitted model: ladders degrade the technical duplicate (after Miller et al.; §2) in steps, on the benchmark's own targets (post hoc; §4.2, App. F, Table A19). Neither MAE nor Pearson-Δ ranks them in order. MAE scores the full technical duplicate worse than the no-change baseline.
  • CA, the primary score, is from Viñas Torné et al. and was chosen before the benchmark outcomes (§2). On this check it is "PARTIAL (4/5)": it passes four of five ladders. The fifth, cell-count reduction, raises amplitude, which CA rewards. Rescaled to the technical duplicate's size, its rungs fall into the right order in all four cell types (App. F).
  • Cosine CA passes every applicable ladder on both constructs' targets. We recommend it for new benchmarks (§2, §6).
  • Cohorts. Cosine CA, CA and MAE get identical verdicts on the original 197 pairs and the expanded benchmark's 399 (Supp. S28).
  • In the main text, Pearson and MAE remain for fixed predictions (§4.1, §4.3). There they show a change of target, not of skill (Supp. S11).

W1, Q1. SB ratios shrink on top response genes (now App. U and Supp. S43)

Response space changes the size, not the sign (Supp. S43). The fMLP–ridge comparison does not inherit the dependence (last answer below).

  • Selecting genes can raise agreement under both splits.
  • On each perturbation's 50 or 200 most-changed response genes, the random − construct difference in agreement is 0.093–0.136 and 0.046–0.106 SB, positive in every cell type.

Q1. "Distinguish response-profile concordance from model-scoring results"

They are now in separate sections: §3 compares constructs' responses with no model, §4 scores fixed predictions, and §5 holds the trained models.

Q1. Metric and response-gene sensitivity of the central paired comparison

Both were assessed. The comparison is the fMLP–ridge gap change (D at matched cell counts): how much more the fMLP loses than ridge when the target moves from the partner construct's response to the scored construct's.

  • The metrics you cite (Supp. S13). As a sign check, the same frozen predictions were rescored with predeclared metrics built to isolate perturbation-specific signal. The comparison passes four of six primary metrics (centred Pearson after Systema, MAE on the top 200 genes, Spearman-Δ, CA) and reverses under none. This check asks whether the sign is stable, not whether a metric ranks skill.
  • Response genes (Supp. S13). Restricted to the top 200 response genes of the pooled evaluation profile, the comparison stays significantly positive under MAE in all four cell types.
  • CA (App. G). On the expanded benchmark, CA alone of twelve predeclared scores passes calibration; its readout of D was designated after the MAE results for D were known. At fixed hyperparameters, expanded benchmark (399 pairs), D is 0.0237 [0.0158, 0.0325] under CA, significantly positive in 4/4 cell types (Tables A22–A23).

Q2. What justifies the 0.005 MAE and 0.02 Pearson thresholds?

Both were fixed before the fMLP was trained, and the revision anchors them to retraining noise.

  • Timing. Both belong to the registered fMLP endpoint (Supp. S5), specified before the fMLP was trained and after ridge's and Perturb Mean's scores on the scored construct were known (Reproducibility statement).
  • Anchor. Retraining noise is the 95th percentile of the difference in the fMLP − ridge gap between two disjoint three-seed ensembles. The 0.005 MAE threshold is 5–20 times that noise, and the 0.02 Pearson threshold 3.5–12 times (Table S25). The anchoring is post hoc and changes no verdict.

Q2. Absolute model scores and paired changes together

The table gives both for the benchmark (CA; 185 pairs, the original 197 minus genes with a third construct; Table A5). Each row lists three CA scores:

  • A: standard evaluation (trained on every construct, scored on the gene-pooled target);
  • B: the same training, scored on the scored construct's target;
  • C: construct-aware evaluation (scored construct withheld, same target as B).

Target change is the step from A to B. Leakage is B − C (up to rounding), a paired difference in which training alone differs.

ModelCA: A / B / CLeakage [Holm interval]
State0.5597 / 0.5036 / 0.4831+0.0204 [0.0037, 0.0410]
Perturb Mean0.5756 / 0.5092 / 0.4870+0.0222 [0.0108, 0.0372]
Linear-PCA0.5422 / 0.4999 / 0.4855+0.0144 [0.0056, 0.0276]
Ridge0.5185 / 0.4984 / 0.4972+0.0012 [0.0005, 0.0021]
CPA0.5029 / 0.4931 / 0.4923+0.0009 [−0.0015, 0.0032]
biolord0.5383 / 0.5061 / 0.4825+0.0236 [0.0064, 0.0448]
No-change baseline0.4935 / 0.4914 / 0.4914—
Technical duplicate0.6897 / 0.6490 / 0.6490—

The biolord row is outside the predeclared family (Supp. S25). In points, every construct-aware CA margin is bounded at 7.4 points or less (post hoc; Table A8).

For the fMLP–ridge pair of the submission (MAE; absolute scores per target in Tables S12 and S17; post hoc summary in Supp. S12): on the partner construct's targets, the fMLP's lead over ridge is significant in all four cell types (0.0178, 0.0064, 0.0125 and 0.0146 MAE). On the scored construct's targets, it is significant in none (0.0037, −0.0062, −0.0012 and −0.0049). The gap change is 0.56–0.78 of the fMLP's own worsening.

Q2. A changed gap, a ranking reversal, and practical importance

These are three separate claims, each with its own cohort and score.

  1. A changed gap, at fixed hyperparameters. D, the fMLP–ridge gap change, is significantly positive in 4/4 cell types under MAE (original 197 pairs; Supp. S10) and under CA (expanded benchmark; App. G). The same-construct control is ≤ 0 by point estimate in 4/4 (Table S13; Part 3). Under tuning on constructs withheld from training D is not significant; under conventional tuning it persists (App. G).
  2. A ranking reversal, under CA (expanded benchmark; App. M, Table A36).
    • Any flip. The ranking-change test, meaning at least one flip, is retained by Holm. The fMLP–ridge pair is one of the flips: ridge − fMLP goes from −0.020 [−0.035, −0.007] in standard to +0.009 [0.006, 0.013] in construct-aware evaluation (full cell count).
    • The top model. It changes from Perturb Mean to ridge (§5.1). Ridge's lead is +0.008 [0.001, 0.015] (conventionally tuned, full cell count; retained by FDR control, not Holm).
    • Perturb Mean's standard advantage collapses under both scores (cosine CA 0.7892 → 0.5020 on 185 pairs, chance 0.5; Table A7).
  3. Practical importance, in points under CA.
    • 185 pairs: State's margin is 33.8 points in standard and −5.2 in construct-aware evaluation. Of the drop, 26.0 points are target change and 13.0 are leakage (Fig. 3A, Table A9).
    • 177 pairs of the expanded benchmark whose scored construct responds (full cell count): removing it from training removes 85% of Perturb Mean's margin and 8% of ridge's (§5.3, App. H).
    • 399 pairs of the expanded benchmark (fixed hyperparameters, half cell count): ridge alone of four models has a significantly positive construct-aware margin (§5.1; Tables A15, A39).

Q2. Four positive matched MAE intervals, with K562 alone above 0.005; the Pearson reversals

Each interval excludes zero, so the gap changes in every cell type. Pooled over cell types, D also excludes the threshold.

  • The four intervals rest on 57, 60, 49 and 31 genes. Pooled, with shared genes modelled, D is 0.0113 [0.0068, 0.0158] (Supp. S7, Table S20).
  • On the expanded benchmark, D clears the threshold in K562 and HepG2 (Supp. S12).
  • The submission's mean-Pearson reversals remain descriptive (Supp. S10). The tested reversal of the same pair is its CA flip (item 2).

W2. "Difficult to follow … substantial restructuring"

We rewrote the main text from the submission: terms and design now precede results.

  • Order. §2 gives data, targets and scores before the results (§3–§5). §5 holds the benchmark (§5.1), transfer across cell types (§5.2), and efficacy and response shape (§5.3).
  • Terms before results. §1–§2 define the two constructs, the two evaluations, target change, leakage and the points scale. Control floor, known-skill ladder and construct recall are defined in the subsections that report them (§3.3, §4.2, §5.2).
  • Appendix. Table A1 maps each main-text claim to its appendix section and tables. Each appendix section opens with Why / Result / Meaning.
  • Jargon. The phrase you quoted is gone from the main text and appendix. Table A2 defines the terms and symbols, and the main text has no capitalised verdict labels.

The submission's registered MAE and Pearson endpoints are unchanged (Supp. S10). The general response (Part 2, "What changed") lists the changes.

W2. "Tables A7 and A12 … deserve more direct main-text treatment"

In the main text, the central paired comparison is now the benchmark: a subsection, two figures and a table. Your Tables A7 and A12 are now Tables S13 and S18 (Supp. S5, S10).

  • The main-text comparison (§5.1). Each of six models is trained twice, with and without the scored construct. Both arms are scored on the same cells of the scored construct, so training alone differs. Table 1 gives this paired difference, leakage, for State in each training seed. Fig. 3A splits State's standard margin into target change and leakage. Fig. 1C shows both margins of every model.
  • Why the fMLP–ridge comparison moved. Its registered endpoint is MAE, and MAE does not rank known skill in order (§4.2).
  • Its CA readout (App. G). At fixed hyperparameters, expanded benchmark (399 pairs), D is 0.0237 [0.0158, 0.0325] under CA, significantly positive in 4/4 cell types (designated after the MAE results; Tables A22–A23).

Q3. The central paired experiment

In short: two models, two targets of one gene, one construct in training, one withheld.

  1. Pairs. A pair is a gene in a held-out cell type with two constructs, each with at least 40 held-out cells. The partner construct also needs at least 40 cells in each training cell type. Selection uses identifiers and cell counts alone (§2).
  2. Training. Models are fitted on the three other cell types. The scored construct is removed from all training data before normalization or fitting; the gene stays in training through its partner construct (Supp. S5).
  3. What each model receives. Ridge and the fMLP (mean of three seeds) receive the gene's GenePT embedding and no construct input, so each gives one prediction per gene. Predictions are frozen before any target is read.
  4. Targets. In the held-out cell type, the two constructs' cells are cut to the same number, and each set forms a target. The remaining partner cells form a second partner target.
  5. Contrasts. The gap is MAE(fMLP) − MAE(ridge) on one target. D is the gap on the scored construct's target minus the gap on the partner's: does an unseen construct change the comparison? The same-construct control L is that difference between the two partner targets: does resampling one construct change it?

What the two tables establish. Table S13 (your A7): D is positive with every adjusted interval above zero in all four cell types, and L is ≤ 0 by point estimate in all four. Table S18 (your A12): on full-cell-count targets the gap change is again positive in all four, each adjusted interval above zero.

Q3. The 57/60/49/31 genes

They are what the eligibility steps of item 1, plus a GenePT embedding, leave in each held-out cell type (Table S26). The main drop comes from the training minimum in K562 and from the held-out minimum elsewhere. Lowering both minimums to 20 gives the expanded benchmark of 399 pairs and 113 genes (§2).

Q3. Overlap across cell types

The pairs share genes, and inference accounts for it. The original 197 pairs cover 67 genes, with the same construct pair wherever a gene occurs (Supp. S12). Gene effects explain about three quarters of D's variance, so the pairs are worth about 77 independent ones. The pooled estimate models shared genes and still excludes zero and the 0.005 threshold (Supp. S7). The benchmark's intervals resample its 62 genes, not pairs.

Q3. Which population

Multi-construct genes: 5.6–5.7% of genes and 9.8–14.1% of cells per screen (§1). For them, standard evaluation does not test generalization to an unseen construct (§6). For single-construct genes, about 94% of core-screen genes, withholding the construct withholds the gene (§7). There the paper recommends a gene-disjoint evaluation (§6, step 5; Supp. S22).

Reviewer_pj98OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper points out that perturbation benchmarks name a gene but score against a response realized by a specific reagent, and that in the four core CRISPRi panels (K562, RPE1, Jurkat, HepG2) every one of the 2,053 shared genes reuses at least one complete dual-sgRNA cassette across contexts, so context holdout is not construct holdout. With matched cell counts, alternative constructs for the same gene are far less concordant than random splits (SB difference 0.15–0.20), and this recurs across panels and in external KO/CRISPRi libraries. Holding predictions fixed, changing the realized target changes absolute scores (including released State predictions). Under global construct exclusion, an fMLP incurs a larger transfer penalty than ridge, flipping mean-Pearson ordering, and State trained from scratch loses 0.10–0.13 Pearson on withheld constructs in RPE1 and Jurkat.

Critical strengths

1- The core observation — that the standard cross-context holdouts on Replogle/Nadig never actually hold out the construct — is simple, checkable, and I hadn’t seen it stated. The matched-cell random-vs-construct split design is a clean way to show the target itself moves.

2- The evaluation contract in Section 6 (declare target realization, declare which constructs are in source data, report both targets with paired uncertainty) is practical and the kind of thing benchmark authors could actually adopt.

Critical weaknesses

1- The relative-performance effects are small and fragile. Only K562 clears the preregistered materiality margin for D; cohorts are 31–60 genes; the released-State effect is driven by one or two genes per fold (top gene is 28–50% of the sum, and Jurkat loses resolution after dropping two). The paper is appropriately careful about this, but “construct transfer changes relative evaluation” is stated in the abstract more confidently than the appendices support.

2- The mechanism is unidentified. Withheld constructs have systematically fewer cells and the paper cannot separate efficacy, preparation, and exposure. Appendix A proves the latent decomposition isn’t identified, which is honest, but it means the practical recommendation is “declare everything” rather than “here’s how to fix it.”

Questions for authors
  • In Table A8, the withheld construct has 33–53 fewer cells on average in three of four folds. Does the matched-n_P analysis fully remove this, or could library-size/UMI differences still drive the penalty?

  • The fMLP vs ridge gap change is ~0.01 MAE. What does that correspond to in practice — would it change a leaderboard?

  • Is the finding specific to dual-sgRNA cassettes? The single CRISPRa case (IER5L) shows no construct effect at higher depth.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer pj98

Thank you for the careful review, and for naming the paper's two central points: holding out a cell type is not holding out a construct, and the reporting contract of §6. Table A59 gives each claim's timing (predeclared or post hoc) and its error control. Cosine CA is post hoc throughout. The main text was rewritten for readers outside biology (general response, Part 2, "What changed").

In brief: yes, construct-aware evaluation changes a leaderboard (Q2); §6 gives a five-step fix with its files available now (W2); and the abstract no longer rests on relative MAE (W1).

W1 (abstract vs evidence: "construct transfer changes relative evaluation").

We agree the abstract should not rest on the fMLP–ridge MAE contrast D, and it no longer does.

  • The abstract as it now reads. The relative-MAE sentence is gone. The abstract states CA results on named populations: State's margin moves from +33.8 to −5.2 points between standard and construct-aware evaluation. Five of six models significantly outperform the no-change baseline in standard evaluation, but none retains a significant advantage under construct-aware evaluation (185 pairs). D stays unchanged in Supp. S10, with the materiality results you cite.
  • The relative contrast the abstract now carries. Where the scored construct responds (177 pairs, expanded benchmark, full cell count), withholding it from training removes 85% of Perturb Mean's margin, against 8% of ridge's (§5.3): Perturb Mean keeps 0.15 [0.05, 0.27] of its margin (App. H). This loss survives Holm over all 522 registered tests (App. X).
  • D and the 0.005 MAE threshold. Pooled over cell types on the original cohort, D is 0.0113 [0.0068, 0.0158] MAE: the interval excludes zero and the threshold, and 17 genes must be removed before it includes zero (Table S20; Supp. S7). Under tuning on constructs withheld from training D is not significant; under conventional tuning it persists (App. G). Per cell type on that cohort, K562 alone reaches the threshold, as you say (Supp. S5); on the expanded benchmark D clears it in K562 and HepG2 and is reproduced on the added genes alone in all four cell types (Supp. S12).
  • D under CA. At fixed hyperparameters, expanded benchmark (399 pairs), D is 0.0237 [0.0158, 0.0325] under CA, significantly positive in 4/4 cell types, on the one predeclared score that passes calibration there (App. G; this CA readout was designated after the MAE results were known).
  • "Cohorts are 31–60 genes." The expanded benchmark, predeclared before outcomes, raises the cohorts from 57/60/49/31 genes per cell type to 107/104/106/82 (Table S26; power: App. N): 113 genes and 399 pairs.
  • Released State "driven by one or two genes per fold." Pooled over cell types, the released-State excess is 0.0139 [0.0050, 0.0228], and 15 genes must be removed before its interval includes zero (Table S20); §4.3 states the per-cell-type concentration. The evidence for State does not rest on released predictions. Trained from scratch, its leakage is a CA result (cosine CA: +0.0391 [−0.0018, 0.0827]; Table A12): under CA it is significantly positive in each of three training seeds and positive in all 12 seed × cell-type estimates (Table 1).

W2 (efficacy, preparation, exposure; "declare everything" vs "here's how to fix it").

The fix needs no mechanism, and §6 gives it as a five-step protocol. The split files, the construct-reuse audit script and the code are available at the anonymous repository https://anonymous.4open.science/r/construct-aware-eval-4B44/.

  1. Record full construct identifiers and audit training–test reuse with our script.
  2. For multi-construct genes, withhold each scored construct from all training data before preprocessing, keeping the partner construct.
  3. For new benchmarks, evaluate cosine CA against the known-skill ladder before using it for model comparison (§4.2).
  4. Report both standard and construct-aware margins, and separate target change from leakage (Table A9).
  5. For single-construct genes, use a separate gene-disjoint evaluation (Supp. S22).

On mechanism:

  • Efficacy: separated from response shape. Of the 85 percentage points Perturb Mean loses, 76 [60, 90] come from pairs where the partner construct does not respond and 9 [1, 18] from pairs where both respond (§5.3; App. I).
  • Exposure: matched. For D, cell counts are matched exactly and UMIs in a predeclared control; the 85% loss is measured on one fixed target (Q1).
  • Preparation: not separated. The causes, "including promoter choice and construct preparation, remain unresolved" (§7).
  • Latent decomposition. Supp. S4 (your Appendix A) still holds: split statistics alone do not identify a latent gene-level share.

The protocol depends on none of these: it fixes which construct defines the target and which constructs are in training.

Q1 (does matching n_P remove the cell-count difference, or could library size / UMIs still drive the penalty?).

Neither cell count nor UMIs drives D.

  • Cell count is matched exactly. The partner-construct and scored-construct targets each hold n_P = min(n_U, n_R − 20) cells (Supp. S12). The cell-count differences you cite from Table A8 are now in Table S14.
  • UMIs are matched in a predeclared sequencing-depth control. Downsampling the deeper pool to the other's exact response-gene UMI total keeps 95–99% of D. Its rule passes in all four cell types for both UMI-matching variants (Supp. S12, Table S24).
  • D tracks neither difference. Across the 197 gene–cell-type pairs of the original cohort, D is not associated with the UMI difference (Spearman ρ = 0.05) or with the cell-count difference (ρ = 0.07; Supp. S12).
  • The benchmark results hold the target fixed. Leakage and Perturb Mean's 85% loss compare two training sets on one fixed target, so its cells are the same in both arms (Fig. 4). Target change persists when the gene-pooled target is cut to the scored construct's cell count: for State, 28.8 [14.5, 45.7] CA points against 26.0 (post hoc; §4.1, Table A10).
  • State. For State trained from scratch, at matched cell counts the Pearson penalties keep 82–91% of their full-cell-count values (post hoc cell-count match; App. W).

Q2 (the gap change is ~0.01 MAE: what does it mean in practice, and would it change a leaderboard?).

Yes, on multi-construct genes. On State's released split (459 gene–cell-type pairs, 116 genes), all three baseline pairs flip between standard and construct-aware evaluation: Perturb Mean minus the no-change baseline goes from +0.072 [0.055, 0.091] CA to −0.015 [−0.023, −0.008], so the standard leader ends below the baseline (predeclared; App. M, Table A35).

In MAE itself, a gap change of this size is the difference between a lead and no lead. On partner-construct targets fMLP's MAE lead over ridge is significant in all four cell types; on the scored construct's targets it is significant in none. The gap change is 1.1–1.8 times the MAE gap between ridge and Perturb Mean in K562, RPE1 and Jurkat, and 6.3–10.6% of fMLP's absolute MAE on partner-construct targets at full cell count (post hoc; Supp. S12, "Practical size").

  • The ranking changes under our protocol, for State too. On the expanded benchmark (399 pairs, conventionally tuned) ridge flips against each of the other three models, at half and at full cell count. The ranking-change test, at least one flip, has p = 6.0×10⁻⁵ and is retained by Holm (App. M, Table A36; Supp. S35). On the six-model benchmark (185 pairs), the plan's ranking analysis lists State vs ridge as a robust reversal under CA (Supp. S25).
  • The top model changes under CA. The leader moves from Perturb Mean to ridge: ridge − Perturb Mean is −0.105 [−0.130, −0.082] under standard evaluation, and ridge's construct-aware lead is +0.008 [0.001, 0.015] (conventionally tuned, full cell count; retained by FDR control, not Holm; Table A36). Under CA the change persists where the scored construct responds (App. J).
  • Your pair flips. Ridge − fMLP goes from −0.020 [−0.035, −0.007] to +0.009 [0.006, 0.013] CA at full cell count (Table A36). The fMLP − ridge flips exceed retraining noise, at 2.2–11.8× the seed standard deviation (Supp. S35).
  • Under both scores, the standard leader's advantage collapses. Between the two evaluations at full cell count, the ridge − Perturb Mean gap swings by 0.79 [0.71, 0.90] of Perturb Mean's standard margin under CA and by 0.879 [0.729, 1.044] under cosine CA (App. M, Table A21).

Q3 (is the finding specific to dual-sgRNA cassettes? The CRISPRa case, IER5L).

No, not for CRISPRi.

  • External single-guide screens. In CRISPR KO and CRISPRi screens, guides of one gene disagree more than equal-cell-count resamples of the same guide. Both CRISPRi datasets significantly exceed their control floors; neither KO dataset does (§3.3; Tables S2–S4).
  • Independently delivered guides. In a CD4+ T-cell CRISPRi screen with two guides per gene on separate vectors, mostly at the same transcription start site, the scored guide's training mean outperforms its partner's when a donor is held out: +0.026 [0.023, 0.030] CA, significantly positive in all 12 contexts (4 donors × 3 conditions; exploratory; App. T, §7).
  • IER5L. It is the CRISPRa dataset's one distinct-spacer pair: 47 of 48 genes use the same spacer in both positions (Supp. S3). CRISPRa is therefore untested rather than negative (§7).

In short: standard margins do not carry over to an unseen construct, the top model changes under CA, the standard leader's advantage collapses under both scores, and §6 gives the fix, with its files available now. Thank you again.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers and the AC: what the paper establishes

Since submission, we added a six-model benchmark (185 pairs), a four-model extension (399 pairs), and known-skill metric checks. Three findings now support the title's claim. Margins are in points: 0 is the no-change baseline and 100 is the technical duplicate. CA (centroid accuracy) is the primary score; cosine CA uses cosine distance and ignores amplitude.

  1. Margins earned under standard evaluation do not carry over to an unseen construct. State is trained from scratch with its released zero-shot recipe. Its CA margin is +33.8 points under standard evaluation. It falls to −5.2 points under construct-aware evaluation, which withholds the scored construct from training, keeps its partner construct, and scores predictions against the scored construct's own response. Under CA, five of six models significantly outperform the no-change baseline in standard evaluation, but none retains a significant advantage under construct-aware evaluation (185 pairs; §5.1, Fig. 1C).
  2. Perturb Mean, the gene's training mean, owes its margin to having seen the scored construct. Where the scored construct responds, withholding it removes 85% of Perturb Mean's margin and 8% of ridge's (CA, target fixed; 177 pairs of the expanded benchmark, full cell count; §5.3, Fig. 4, App. H).
  3. A larger benchmark separates one model from the baseline. On the expanded benchmark (399 pairs, 113 genes), with the scored construct withheld, ridge alone of four models is significantly above the no-change baseline: +0.0057 [0.0015, 0.0102] CA (fixed hyperparameters, half cell count; §5.1; Table A39).

Table A59 gives each claim's timing (predeclared or post hoc) and its error control. All 522 registered tests, 196 of them post hoc, are corrected together, and Holm retains 294 (Table A62). The 294 include the 85% loss (§5.3), the scored construct's transfer advantage over its partner (§5.2) and the response-shape difference between a gene's two constructs (§3.1; Table A63).

Evidence on the shared questions

  • Reuse is built into the data and the released splits. For each of the 2,053 genes shared across the four core CRISPRi screens, at least one identical construct appears in all four cell types. The splits released with State and PerturBench keep every scored construct in training (§1, App. B, G).
  • The drop has two sources. For State under CA, 26.0 points come from target change and 13.0 from leakage (185 pairs; Fig. 3A, Table A9). Target change is the margin gained by scoring the gene-pooled response instead of the scored construct's response, with predictions fixed. Leakage is the margin gained by training on the scored construct, on the same target. State's target change is significant under cosine CA too.
  • Leakage holds across models (185 pairs). It is significantly positive under both CA and cosine CA, and under four amplitude-matching methods, for Perturb Mean, Linear-PCA and ridge (Tables A5, A11). Under CA it is also significantly positive for biolord (Table A5) and for State, in each of three training seeds (Table 1). State's leakage is a CA result: under cosine CA its three-seed estimate is +0.0391 [−0.0018, 0.0827] (Table A12), and the CA effect depends on prediction amplitude (Table A11).
  • The top model changes under CA. Withholding the scored construct moves the lead from Perturb Mean to ridge (§5.1; App. M, Table A36). The ranking-change test, meaning at least one flip, is retained by Holm. On the expanded benchmark, ridge's lead is +0.008 [0.001, 0.015] (conventionally tuned, full cell count; retained by FDR control, not Holm). Perturb Mean's standard advantage collapses under both scores: on 185 pairs its cosine CA falls from 0.7892 to 0.5020 (chance 0.5; Table A7).
  • Even the partner construct's own measured response scores near the no-change baseline, under both scores. Rescaled to the technical duplicate's amplitude, it scores −3.5 [−11.1, 1.8] points under CA as a prediction of the scored construct. The gene-pooled response, which contains the scored construct's cells, scores 78 (185 pairs; Fig. 3B, Table A20).
  • The contrast holds where both constructs knock the gene down. On those 74 of the 185 pairs, the standard margins of State and Perturb Mean are significantly positive under both scores, and no construct-aware margin is (§5.1, Table A14).
  • Transfer follows the construct. On State's released split (459 pairs, 116 genes), the scored construct's training mean beats the partner construct's by 0.097 [0.079, 0.116] CA. The advantage is significant in every cell type under both scores (§5.2, App. K, Table A21).
  • On State's released split, all three baseline pairs flip between the two evaluations. Perturb Mean minus the no-change baseline goes from +0.072 [0.055, 0.091] to −0.015 [−0.023, −0.008] CA (same pairs; Table A35).
AuthorsOpenReview ↗

General response to all reviewers and the AC: the revision

What changed

  • Main text. §1–§7 are rewritten for readers outside biology and end on p. 10. §1–§2 define the terms, design and scores before the results (§3–§5). §3 gives model-free evidence that constructs of one gene differ. §4 scores fixed predictions against different targets and checks the scores. §5 holds new evidence: the benchmark (§5.1; Fig. 3, Table 1), transfer (§5.2), and efficacy and response shape (§5.3, Fig. 4). §6 gives a five-step protocol; §7 gives scope and conclusion.
  • Abstract. It is rewritten around the three results in Part 1 and names its score, CA.
  • Primary score. CA is the primary score, chosen before the benchmark outcomes; cosine CA is post hoc, and we recommend it for new benchmarks (§2, §6; App. F).
  • §4.2, the known-skill ladder: new. The scores are checked on predictions of known skill: the ladder degrades the technical duplicate step by step (App. F, Table A19). Cosine CA passes every applicable ladder on both construct targets. CA passes four of five ladders ("PARTIAL (4/5)"); the exception is cell-count reduction, which raises amplitude. MAE and Pearson do not order known skill.
  • §3.3, external screens: kept from the submission, rewritten. External single-guide CRISPR KO and CRISPRi screens show greater response disagreement between guides of one gene than between equal-cell-count resamples of the same guide. Both CRISPRi datasets significantly exceed their control floors (Supp. S1).
  • §4.3, State under fixed predictions: the submission's §4.2, rewritten. For the released State predictions, squared Pearson-score disagreement across a gene's two constructs exceeds same-construct resampling by 0.0139 [0.0050, 0.0228] pooled (Table S20; Supp. S23). The interval stays above zero until the top 15 genes are removed. State trained from scratch without the scored construct scores significantly lower in Pearson on that construct than on its partner in RPE1 and Jurkat (App. W). Here the Pearson changes reflect the target, not skill (Supp. S11).
  • The fMLP–ridge comparison moved out of the main text. App. G reports it under CA. D is how much more the fMLP loses than ridge when the target moves from the partner construct's response to the scored construct's. At fixed hyperparameters, expanded benchmark (399 pairs), D under CA is 0.0237 [0.0158, 0.0325], significantly positive in all four cell types. This CA readout was designated after the MAE results were known. Supp. S10 keeps the submission's registered MAE and Pearson endpoints unchanged.
  • One glossary. The main text and the appendix use one set of terms. Table A2 defines them; the supplement's reading key glosses its symbols. The submission's "withheld construct" is the scored construct, the one kept in training is the partner construct, and "global construct exclusion", scored on the scored construct's own response, is construct-aware evaluation.
  • Appendix and supplement. Each of App. A–X opens with Why / Result / Meaning, and Table A1 maps every main-text claim to its evidence. Supp. S1–S23 hold the submission's analyses and their robustness checks; Supp. S24–S46 give each appendix section's full settings. For labels cited in the reviews: submission Tables A7 and A12 are now Tables S13 and S18, Table A8 is Table S14, Appendix A is Supp. S4, Appendix B is App. U with Supp. S43, and Appendix J is App. W.

Cohorts and scope

  • Cohorts and power. The six-model benchmark has 185 pairs (62 genes); the expanded benchmark has 399 pairs (113 genes). With 113 genes, construct-aware comparisons detect model differences of 0.006–0.011 CA at 80% power (App. N). On 185 pairs, every construct-aware CA margin is bounded at 7.4 points or less (Table A8).
  • Population. Direct construct comparisons apply to multi-construct genes, which are 5.6–5.7% of genes and 9.8–14.1% of cells per screen (§1). For single-construct genes, standard evaluation tests transfer of a seen construct to a new cell type; gene-disjoint evaluation tests unseen genes (§6, step 5).

Resources. The split files, the construct-reuse audit script and the code are available at the anonymous repository https://anonymous.4open.science/r/construct-aware-eval-4B44/.

What is established. A gene label does not specify the measured response a model is scored on. Standard and construct-aware margins differ by target change plus leakage. Under CA, five of six models significantly outperform the no-change baseline in standard evaluation, but none does under construct-aware evaluation (185 pairs). Where the scored construct responds, withholding it removes 85% of Perturb Mean's margin and 8% of ridge's. A benchmark should therefore state which construct defines its target and which constructs are in its training data (§6).

We thank the reviewers and the AC for their careful reading and their questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.