acceptodds
Withdrawn from ICLR 2027

A Pitfall in the Evaluation of Gene Perturbation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Gene perturbation models aim to predict transcriptional responses to unseen genetic interventions, but comparisons rely on heterogeneous evaluation metrics and differentially-expressed (DE) gene subsets. We investigate the consequences of these choices using eight methods across six single-gene perturbation datasets under a unified evaluation protocol. Varying the number of top DE genes reveals substantial instability in Precision-based rankings, with the winning method changing 86.7% of the time, while rankings based on the rest of the metrics remain comparatively stable. Investigating this discrepancy exposes that repeating a single predicted expression profile across cell-level rows, a standard practice for many methods, can inflate significance estimates and produce excessively large predicted significant-DE sets. Specifically we found that 67.1% of tested perturbation–gene pairs pass the significance threshold despite some of them hav- ing minuscule effects. We derive an inflation factor showing that replicated predictions can amplify the standardized Mann–Whitney statistic even after correction for tied ranks, without adding independent information. This artifact changes Precision mainly as the number of DE genes increases because for smaller cutoffs, most of the genes are already significant due to the effect-based ordering. Taking this pitfall into account in our benchmark comparison, we find that a random forest using Gene Ontology features achieves the best overall results.

Primary area:
datasets and benchmarks
Keywords:
gene perturbation models, evaluation, benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UdwnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors argue that a common problem exists in evaluating perturbation prediction. The DE gene sets are not precisely predicted.

Critical strengths

The results are interesting with comprehensive statistical studies.

Critical weaknesses

I think the proposed questions are very interesting, and the others do introduce a statistical framework to address them; however, I doubt whether it can really affect practical usage, as well as the leakage from using Gene Ontology features.

  1. The paper is presented in a form more like an empirical study rather than a rigorous research paper. There are many approaches we can use to evaluate the quality of perturbation predictions, and Pearson and MSE are two naïve choices. Several researchers have explored better metrics, but this manuscript does not discuss them.
  2. The methods selected in this study are outdated; updated perturbation prediction models, such as STATE and X-cell, are not discussed. The reasons could come from the low performance of selected models and thus it is not reliable.
  3. The analysis figure in Figure 1 is very weird. The curve is hard to read and understand. Here, I cannot find obvious differences across methods, and I cannot identify the best methods directly.
  4. The datasets are not precisely assigned. The VCC has more information in public training datasets. The authors also need to consider Tahoe and larger datasets.
  5. MWU test is just one option for selecting DE gene effects, and since the studies are based on non-parametric and unpaired datasets, the p-value could be altered very easily even with a small change; thus, the authors need to use simulation data to justify their selection first.
  6. Overall I think this paper is more suitable for conference such as AISTAT or Bioinformatics, rather than a ML-centered conference.
Questions for authors

Please see the weaknesses.

Reviewer_JKDrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper deals with an important task: benchmarking eight methods and baseline across six single-gene perturbation datasets in order to discern the role of different cutoffs for DE genes within Gene Perturbation Models. In other words, it studies how the choice of evaluation protocol affects the ranking of gene perturbation models (GPMs) on unseen single-gene perturbations. This paper considers 8 methods as shown in Table 1, and 6 CRISPRi datasets as shown in Table 2.

Critical strengths
  • The benchmark task itself is meaningful and interesting, such a unified protocol over six datasets and eight methods is a worthwhile controlled study
  • The considered 8 methods and 6 datasets are somewhat convincing.
  • The structural asymmetry between overlap and precision is a clean and checkable observation about metric design.
Critical weaknesses
  • This paper is presented very badly: 1) Figure 1 is totally unreadable and inacceptable. 2) Table 1 (right side) is out of margin. 3) Figure 3 is sparsely presented, leaving a lot of empty space, maybe using wraptable can be helpful.
  • In the experiment part, there is no ablation study to isolate different modules.
  • The derivation assumes no ties in the observed group and a single tie group, which is limited.
  • Does the finding from this paper can be simply extended to the evaluations of other related biological foundation models?
Questions for authors

See above.

Reviewer_HbP7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares eight gene perturbation methods on six datasets and studies sensitivity to the number of evaluated DE genes. In its evaluation pipeline, a perturbation-level point prediction is repeated across cell-level rows before DE testing. The authors derive an amplification factor for the nominal Mann–Whitney statistic and relate enlarged predicted DE lists to Precision sensitivity. They then report model rankings using small DE cutoffs.

Critical strengths
  1. The tie-corrected replication factor makes the effect of duplicated point predictions on nominal Mann–Whitney significance explicit and checkable.

  2. The common perturbation-disjoint splits across eight methods and six datasets document sensitivity to the DE cutoff.

Critical weaknesses
  1. The rationale for replicating point predictions is insufficient. Section 3.1 motivates replication by matching prediction and observed matrix dimensions, but equal sample sizes are not required by the Mann–Whitney U test, and copying creates no independent information. The paper should justify why this adaptation is appropriate and representative. The submitted adapters add this step, while the official GEARS and RF-GO scoring paths compare condition means without replicated-prediction DE testing. Cell-Eval's official baseline contains a related procedure, but its existence does not establish widespread use in published benchmarks.

  2. The final benchmark does not validate its significance-based metrics. Section 4.2 acknowledges that small N does not correct the p-values; it restricts evaluation to stronger retained effects. Precision and Overlap therefore remain dependent on the affected predicted DE sets. Greater stability across cutoffs does not demonstrate robustness to replication count, for which no controlled sweep is reported. The practical impact on the reported model comparisons needs clearer validation.

Questions for authors
  1. Why is matching prediction and observed matrix dimensions necessary, given that the Mann–Whitney U test allows unequal sample sizes?

  2. Which published benchmarks and softwares apply DE testing to replicated point predictions, and how widely is this replication procedure used?

  3. With all predictions and data fixed, how do DE calls and model rankings change when only the replication count varies?

  4. Do the reported Precision and Overlap rankings persist under an evaluation that avoids treating copied predictions as independent cells?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.