- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how CRISPR construct identity affects genetic perturbation model evaluation. Across multiple datasets, the authors show that different constructs targeting the same gene produce distinct responses and can change model evaluation scores. Under global construct exclusion, the authors further show differential transfer penalties across models and descriptive ranking reversals. The paper argues for benchmarking that is construct-aware and explicitly specifies target realization and construct availability in training data.
Critical strengths
1. An underexplored aspect of evaluating perturbation prediction
The authors show that the gene which is perturbed does not uniquely determine the experimental response. In particular, how construct identity overlaps with training and evaluation can affect scores and sometimes model rankings.
2: Good experimental design
The paper shows how construct identity affects scoring across multiple external CRISPR dataset and cell lines, using careful controls (e.g., matched cell counts, same construct resampling controls, etc.).
Critical weaknesses
The generality of the demonstrated effects on model comparisons is limited The paper convincingly shows how construct identity can alter absolute performance, however, the evidence for broader implications for model comparisons is less established. The main performance analysis compares ridge regression with an adjusted fMLP on relatively small cohorts. The matched-cell count MAE gap changes are statistically supported across contexts, however, only K562 exceeds the prespecified materiality threshold. The State experiments do show sensitivity in absolute performance, but do not show alterations in rankings among modern perturbation models.
Questions for authors
1. Clarification: Significance of changes in model rankings
Could the authors clarify how the findings support adjustments in pratical model selection, rather than differences in transfer penalties? Is there evidence that the ranking reversals you observe are robust under paired uncertainty?
2. Additional analysis: Performance comparisons for State
Given that the from-scratch trained State and predictors are evaluated under the same global construct exclusion framework, could the authors compare State directly against ridge regression and/or fMLP on the shared evaluation targets, with paired uncertainty? Such a comparison would help establish whether the construct realization can materially change conclusions about model selection beyond the primary fMLP ridge comparison.
4 comments
Q1a. "how the findings support adjustments in practical model selection, rather than differences in transfer penalties?"
A difference in penalties matters for selection when it reverses an order, and here it does. Under CA, standard evaluation selects Perturb Mean and construct-aware evaluation selects ridge (§5.1; App. M).
- Training alone reverses the order on one fixed target. Scored on the same construct target, ridge − Perturb Mean is −0.027 [−0.037, −0.018] with the scored construct in training and +0.008 [0.001, 0.015] without it (CA; Table A15; expanded benchmark, conventionally tuned, full cell count; error control in Q1b). On the 185 pairs, the same fixed-target comparison lists Perturb Mean vs ridge as a robust reversal (plan analysis S1b, Supp. S25).
- Tuning selects differently. Conventional and construct-aware tuning select different configurations in all four cell types (Supp. S8).
- Standard evaluation credits two gains that a new construct does not supply. Target change comes from scoring the gene-pooled response instead of the construct's own. Leakage comes from having the scored construct in training. Under CA they are worth 26.0 and 13.0 points to State (Fig. 3A, Table A9; 185 pairs). State's target change is significant under both scores. Its leakage is significant under CA; under cosine CA its interval includes zero, at +0.0391 [−0.0018, 0.0827] (three seeds; Table A12).
- The gains differ by model. Leakage is 14.1 CA points for Perturb Mean and 0.7 for ridge, significant for both under CA and cosine CA; target change is 30.6 points for Perturb Mean (Tables A5, A9).
- Both gains move the ridge–Perturb Mean gap. At full cell count on the expanded benchmark, the plan's verdict under CA is that target change carries the swing; leakage is 0.31 [0.24, 0.38] of it (App. D). Under cosine CA the label is that both contribute (Table A21).
- The loss differs by model. Where the scored construct responds, withholding it removes 85% of Perturb Mean's margin and 8% of ridge's (§5.3, App. H; expanded benchmark, 177 pairs, full cell count).
In practice: report both standard and construct-aware margins, and select on construct-aware evaluation when the model will be used on new constructs. §6 states that the training split should match the generalization task (steps 2 and 4).
Q1b. "Is there evidence that the ranking reversals you observe are robust under paired uncertainty?"
Yes. A flip requires a pair's gap to be significant under both evaluations, with opposite signs (App. M). Ridge flips against Perturb Mean, Linear-PCA and the fMLP, at half and at full cell count. Each interval in Table A36 is on the difference between two models over the same gene–cell-type pairs, not on each model separately (CA; expanded benchmark, 399 pairs; conventionally tuned; 95% gene-cluster bootstrap). At full cell count:
| Pair | Standard | Construct-aware |
|---|---|---|
| ridge − Perturb Mean | −0.105 [−0.130, −0.082] | +0.008 [0.001, 0.015] |
| ridge − Linear-PCA | −0.033 [−0.051, −0.018] | +0.012 [0.007, 0.018] |
| ridge − fMLP | −0.020 [−0.035, −0.007] | +0.009 [0.006, 0.013] |
- The ranking-change test, at least one flip, has p = 6.0×10⁻⁵ and is retained by Holm (Supp. S35; App. X).
- Ridge's lead in the first row, +0.008 [0.001, 0.015] (conventionally tuned, full cell count; retained by FDR control, not Holm), is a CA result.
- At full cell count, Perturb Mean's standard lead and ridge's construct-aware lead are both significant under conventional tuning and with fixed hyperparameters; under the specificity floor and under construct-aware tuning, the ridge − Perturb Mean flip weakens there (Table A37). At half cell count every configuration repeats the lead change (App. M).
- The fMLP–ridge flips are 2.2–11.8× the seed standard deviation, with the same sign in 3 of 3 seeds (Supp. S35).
- The six-model benchmark agrees under CA. On the 185 pairs, ridge exceeds Perturb Mean under construct-aware evaluation by +0.0102 [0.0021, 0.0195] CA; under cosine CA the difference is +0.0175 [−0.0379, 0.0743] (Supp. S25).
- The reversal in the submission was a descriptive mean-Pearson ordering (Supp. S10). Pearson fails every applicable known-skill ladder (§4.2), so the revision tests reversals under CA, where the same fMLP–ridge pair flips (third row).
- Perturb Mean's advantage collapses under both scores. The swing is 0.79 [0.71, 0.90] of its standard margin under CA (App. M) and 0.879 [0.729, 1.044] under cosine CA (Table A21).
Q2a. "compare State directly against ridge regression and/or fMLP on the shared evaluation targets, with paired uncertainty"
On shared targets, yes. State, ridge and the fMLP are fitted under both evaluations and scored on the same 185 pairs and the same targets (Table A7). The paired evidence the revision prints is listed below. Under CA it pairs each model with the no-change baseline and with its own fit that saw the scored construct; in MAE it pairs State with ridge and with the fMLP.
| Model | Pooled CA, standard → construct-aware | Construct-aware CA margin [90%] | Leakage under CA, points [95%] |
|---|---|---|---|
| State | 0.5597 → 0.4831 | −0.0083 [−0.0151, −0.0015] | 13.0 [4.0, 22.5] |
| Ridge | 0.5185 → 0.4972 | +0.0058 [−0.0014, 0.0138] | 0.7 [0.5, 1.0] |
| fMLP | 0.5214 → 0.4884 | −0.0030 [−0.0075, 0.0014] | 2.0 [1.1, 3.2] |
| No-change baseline | 0.4935 → 0.4914 | — | — |
By pooled score, State leads ridge under standard evaluation and ridge leads State once the scored construct is withheld; the plan's ranking analysis lists the pair as a robust reversal (Supp. S25, "Ranking"). The pooled scores are from Table A7, whose intervals are per model, not paired differences. The margins are from Table A8 (post hoc; 90% for its two one-sided tests): each is a model minus the no-change baseline over the same pairs, on the shared construct target. Leakage is from Table A9 (post hoc): each model's fit that saw the scored construct minus its fit that did not, on that target.
- Paired CA margins in points, standard evaluation (95% intervals; Supp. S25). State has 33.8 [18.2, 48.7] and ridge 12.7 [3.2, 20.8].
- The same margins with the scored construct withheld. Ridge's interval includes zero, at 3.7 [−2.3, 8.1]; State is significantly below the no-change baseline, at −5.2 [−13.8, −0.1].
- Paired leakage under CA (last column). State's and ridge's stay significantly positive with Holm-adjusted intervals (Table A5). State's is significantly positive in each of three training seeds (Table 1; cosine CA: Q1a).
- Paired MAE gap changes, State vs ridge (the submission's design; Table S28, one State seed). State loses more than ridge from the partner construct's target to the scored construct's in every cell type. The gap change is +0.016 to +0.023, above the 0.005 threshold in each, and 0.0188 [0.0126, 0.0251] pooled (Table S20). By point estimate, State has the lower MAE on the partner construct's target and ridge on the scored construct's, in all four cell types (Supp. S12).
- Paired MAE gap changes, State vs fMLP (Table S28). The gap change is positive in all four cell types by point estimate and significant in RPE1, at +0.0101 [0.0033, 0.0184]. Both MAE comparisons are retained by Holm (claims A27 and A28, Table A61).
- Paired add-back contrast (Table A57). Returning the scored construct to training lowers State's MAE on the construct target more than ridge's, by +0.019 [0.006, 0.042] in RPE1 and +0.020 [0.007, 0.036] in Jurkat.
Q2b. "whether the construct realization can materially change conclusions about model selection beyond the primary fMLP ridge comparison"
Yes. Beyond fMLP–ridge, the conclusion changes in each case below.
- Top model. Under CA it changes from Perturb Mean to ridge (Q1b).
- State and ridge. Their order reverses under CA (Q2a).
- State's released split, the released model's multi-construct subset (88 pairs, 42 genes). Whether the released model beats a predictor that learns nothing depends on which construct that predictor recalls. State minus the partner construct's training mean is +0.024 [0.007, 0.042] CA, and State minus the scored construct's training mean is −0.061 [−0.081, −0.043] (Table A33).
- The same split, single-construct genes. There the construct's training mean is the gene's training mean. Pooled over cell types it has CA 0.672 against State's 0.608; State minus this mean is −0.064 [−0.070, −0.058] (§5.2; Supp. S33). It leads significantly in K562, Jurkat and HepG2; State leads in RPE1 (App. K).
On the expanded benchmark (399 pairs, 113 genes; the four models of Q1b), ridge alone has a significantly positive construct-aware CA margin, at +0.0057 [0.0015, 0.0102] (fixed hyperparameters, half cell count; Table A39).
Presentation. The main text was rewritten for readers outside biology, and Table A1 maps each claim to its evidence (general response, Part 2, "What changed").
Resources. The split files, the construct-reuse audit script and the code are available at the anonymous repository https://anonymous.4open.science/r/construct-aware-eval-4B44/.
Thank you again. Your question was whether the construct changes model selection. With six models on shared targets, under CA it changes the ranking, the top model, and State's position against ridge.
I thank the authors for their detailed experiments, which strengthen the paper. Nevertheless, the findings remain limited to relatively small set of multi-construct genes. Additionally, I share reviewer B4kT's concerns with respect to sensitivity to the choice of metric, as well as the substantial changes that have been introduced with regards to core aspects of the paper during the rebuttal period.
In summary, I still find the contribution worthy of acceptance, and maintain my score of weak accept.
Reply to Reviewer tUgJ
Thank you for the careful review and for recognising the controls, which remain in the revision (§3.1, Supp. S10, Table S13). Your central point is whether the construct changes model comparison, beyond absolute scores. The revision's main addition answers it: six models, each fitted under standard and under construct-aware evaluation, on the same 185 gene–cell-type pairs (§5).
Terms, as in the revised PDFs. Standard evaluation keeps every construct in training and scores the gene-pooled response. Construct-aware evaluation withholds the scored construct from training and scores that construct's own response; the gene's partner construct stays in training. CA is centroid accuracy, the primary score (§2). Points run from 0, the no-change baseline, to 100, a technical duplicate. Table A59 gives each claim's timing (predeclared or post hoc) and its error control.
W1. "the evidence for broader implications for model comparisons is less established"
Under CA the ranking changes, and so does the top model.
W2. "compares ridge regression with an adjusted fMLP"
The main comparison now covers six models: State trained from scratch, CPA, biolord, ridge, Perturb Mean and Linear-PCA, with the fMLP as a reference (§2, §5). The fMLP–ridge comparison moved to App. G (CA) and Supp. S10 (the submission's MAE analysis, unchanged). Under CA the fMLP loses more than ridge from the partner construct's target to the scored construct's. At fixed hyperparameters, expanded benchmark (399 pairs), the gap change D is 0.0237 [0.0158, 0.0325], significantly positive in all four cell types (App. G, Tables A22–A23; this CA readout was designated after the MAE results were known).
W3. "on relatively small cohorts"
The revision enlarges the benchmark and reports what it can detect. The comparison exists for multi-construct genes, which are 5.6–5.7% of genes and 9.8–14.1% of cells per screen (§1).
W4. "only K562 exceeds the prespecified materiality threshold"
Three printed results clear the threshold beyond K562. The sentence you cite is the per-cell-type MAE form of the fMLP–ridge gap change D, kept unchanged in Supp. S10.
W5. "do not show alterations in rankings among modern perturbation models"
Under CA, State's order against ridge reverses. The plan's ranking analysis lists State vs ridge as a robust reversal: State leads ridge under standard evaluation, and ridge leads State once the scored construct is withheld (Supp. S25, "Ranking", supp. p. 63). The pooled scores and the paired evidence are in Q2a.