- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GNEval benchmarks gene-network inference using 21 gene representations or association scores from 11 models across 14 Perturb-seq datasets. It compares static embeddings, contextual embeddings, attention-derived scores, and domain-specific methods using GO-based functional relationships and perturbation-associated differential-expression evidence. The authors report markedly different rankings under the two evidence sources. The context-independent group performs better on the GO evaluations, whereas the pooled context-dependent group, comprising contextual embeddings and attention-derived scores, achieves higher regulator-level perturbational enrichment; DE AUROC instead favors the context-independent group. The paper also provides a harmonized preprocessing pipeline, per-dataset results, sensitivity analyses, and computational-cost measurements.
Critical strengths
- Comparing functional and perturbational evidence across 14 datasets provides a useful perspective on the task-dependent strengths of gene representations.
- Detailed preprocessing documentation, per-dataset results, sensitivity analyses, and computational-cost measurements improve the benchmark's transparency and practical utility.
Critical weaknesses
- Ranking metrics use different evaluation sets. GO AUROC/AUPRC and DE AUROC are computed only over each method's selected edges (Appendices F.2/G.3, pp. 21-22), measuring within-network ordering rather than recovery over a shared candidate space.
- Filtering may affect aggregate comparisons. Appendix G.4 excludes combinations with fewer than ten DE-supported edges from AUROC and odds-ratio summaries, while Appendix I acknowledges non-random missingness; method medians can therefore reflect different dataset subsets, without a reported common-dataset sensitivity analysis.
- The contribution of context is not isolated. The static and contextual groups contain different model/readout mixtures, and Geneformer uses V2-316M for static embeddings versus V2-104M for contextual embeddings and attention (Table 5); these differences limit attribution of the group-level results to target-dataset context.
- Baseline inputs are not matched. Contextual methods use 1,000 cells from the full Perturb-seq population (Section 2.3), whereas CS-CORE uses all non-targeting control cells (Appendix D.3), mixing differences in cell composition and sample size with differences between methods.
Questions for authors
- Do the rankings persist when AUROC/AUPRC are evaluated on a shared annotated/testable candidate set, with unavailable scores handled explicitly?
- How do rankings change when low-hit cases are retained where metrics remain defined and methods are compared on the same datasets?
- Does the static-versus-contextual difference persist with identical checkpoints and common represented genes, analyzing attention separately?
- How do contextual methods and CS-CORE compare when both use the same number of non-targeting control cells?
- What edge counts were used for CellOracle in each dataset, and was it included in the network-size and fixed-density analyses?
4 comments
Response to Reviewer Q5DD (2/4)
Response to Reviewer Q5DD (2/4)
Question 2
Question 2: “Filtering may affect aggregate comparisons. Appendix G.4 excludes combinations with fewer than ten DE-supported edges from AUROC and odds-ratio summaries, while Appendix I acknowledges non-random missingness; method medians can therefore reflect different dataset subsets, without a reported common-dataset sensitivity analysis. How do rankings change when low-hit cases are retained where metrics remain defined and methods are compared on the same datasets?”
We thank the reviewer for highlighting the potential selection bias introduced by excluding low-hit cases in perturb-seq-based evaluation. We agree that this is an important concern. The original minimum threshold of 10 DE-supported edges was intended to reduce noise from sparse positive labels. To address this concern, we (1) lowered the minimum number of DE-supported edges from 10 to 1, retaining all cases with defined metrics; (2) included zero-hit cases by assigning DE OR = 0 and DE AUROC = 0.5, with the latter serving as a neutral ranking convention because AUROC is undefined without positive edges; (3) reduced the minimum number of testable regulators for enrichment evaluation from 10 to 1; and (4) extended the analysis to all 14 datasets.
The rankings remain robust after we include 14 datasets with a perturb-seq-based rank correlation of 0.913 across all 21 methods. Importantly, our main conclusion remained unchanged: context-dependent associations showed stronger recovery of perturbation evidence.
We acknowledge that assigning DE AUROC = 0.5 may favor zero-hit methods over those with slightly below-chance performance. For example, scFoundation_C moves from 21st to 9th in DE AUROC and from 21st to 17th in perturb-seq-based rank. We therefore interpret these rankings with caution and emphasize the stability of the overall conclusions rather than exact method positions.
We will include both evaluations and clarify their complementary interpretations in the revised manuscript.
Question 3
Question 3: “The contribution of context is not isolated. The static and contextual groups contain different model/readout mixtures, and Geneformer uses V2-316M for static embeddings versus V2-104M for contextual embeddings and attention (Table 5); these differences limit attribution of the group-level results to target-dataset context. Does the static-versus-contextual difference persist with identical checkpoints and common represented genes, analyzing attention separately?”
We thank the reviewer for identifying the checkpoint mismatch. To address this, we evaluated static Geneformer using V2-104M, matching the checkpoint used for contextual embeddings and attention-derived scores. We directly compared all three representations under the same evaluation protocol, retaining V2-316M static embeddings as a reference. Paired Wilcoxon tests were performed on dataset-level differences, with BH correction across six metrics.
We made two key observations from Table R2:
1. Static representations show broadly consistent performance across checkpoints, as shown by columns (1) and (2).
2. The static-versus-contextual differences largely persist under identical checkpoints, as shown by comparing column (2) with columns (3) and (4). This pattern holds for all three GO metrics (q<0.01), DE OR, and significant-overlap rate. Although the DE AUROC difference reverses direction after checkpoint matching, it remains statistically nonsignificant under both checkpoints.
Table R2. Geneformer representation comparison with matched checkpoints.
| Metric | (1) Static | (2) Static | (3) Contextual | (4) Attention |
|---|---|---|---|---|
| Checkpoint | V2-316M | V2-104M | V2-104M | V2-104M |
| GO Fisher OR | 5.65 (4.0) | 5.47 (4.1) | 2.77 (8.7) | 0.90 (16.2) |
| GO AUROC | 0.678 (5.6) | 0.699 (5.4) | 0.632 (10.7) | 0.532 (14.6) |
| GO AUPRC | 0.081 (5.0) | 0.077 (4.6) | 0.023 (9.2) | 0.003 (15.4) |
| DE OR | 1.44 (12.7) | 1.36 (14.9) | 1.73 (10.6) | 3.83 (3.9) |
| DE AUROC | 0.519 (7.4) | 0.484 (12.9) | 0.492 (9.2) | 0.496 (9.3) |
| Sig. overlap | 0.006 (13.2) | 0.000 (14.2) | 0.022 (9.7) | 0.056 (4.2) |
Values are median performance across datasets, with mean within-dataset ranks among 21 methods in parentheses (lower ranks are better). Results use the top-K evaluation (5k/10k/20k) and GO Jaccard > 0.3.
These results confirm that the GO findings are robust to checkpoint selection, while some perturbation-based comparisons are checkpoint-sensitive. We will use matched checkpoints in the revised manuscript and avoid attributing group-level differences solely to contextualization.
Response to Reviewer Q5DD (3/4)
Response to Reviewer Q5DD (3/4)
Question 4
Question 4: “Baseline inputs are not matched. Contextual methods use 1,000 cells from the full Perturb-seq population (Section 2.3), whereas CS-CORE uses all non-targeting control cells (Appendix D.3), mixing differences in cell composition and sample size with differences between methods. How do contextual methods and CS-CORE compare when both use the same number of non-targeting control cells?”
We thank the reviewer for highlighting the differences in cell sampling. We agree that using the same cells will enable a fairer comparison of contextual methods and CS-CORE. We reran all 11 contextual/attention methods and CS-CORE using identical non-targeting control cells, including all available controls. The two largest datasets GWCD4i and WholeBrain were capped at 12,000 cells due to their large control pools (309,554 and 195,132 cells, respectively), with the cap exceeding the largest uncapped dataset (Norman, 11,483 cells).
The rankings remain highly consistent for GO-based metrics (ρ = 0.972-0.993) and reasonably stable for perturbation-based metrics (ρ = 0.818-0.909) (Table R3). These results indicate that method rankings are largely robust to control-cell sample size, although some perturbation-based rankings are more sensitive. Importantly, all contextual methods and CS-CORE now use matched control-cell inputs, addressing the original concern about baseline comparability.
Table R3. Ranking consistency between 1,000 and all available control cells (12 methods).
| Metric | Spearman ρ | Max. rank change | Method with largest change |
|---|---|---|---|
| GO OR | 0.972 | 2 | GeneCompass_A |
| GO AUROC | 0.986 | 1 | Geneformer_A |
| GO AUPRC | 0.993 | 1 | GeneCompass_A |
| DE OR | 0.909 | 3 | scPRINT_A |
| DE AUROC | 0.818 | 4 | CS-CORE |
| Sig. overlap | 0.860 | 4 | scPRINT_A |
We will update the manuscript to use matched control-cell inputs and clarify the sampling protocol.
Question 5
Question 5: "What edge counts were used for CellOracle in each dataset, and was it included in the network-size and fixed-density analyses?"
We thank the reviewer for this question. Unlike other methods, CellOracle produces sparse networks restricted to TF-target pairs supported by its base GRN inferred from external data. Following the recommended filtering procedure (p < 0.001), the resulting networks contain 2,446-12,634 edges across datasets (Table R4).
Network-size (top-K) analysis: CellOracle is evaluated at all available network sizes (5k, 10k, and 20k edges), plus its full network, and is excluded from network sizes exceeding its available edge count.
Fixed-density and rank-stability analyses: CellOracle is excluded because its maximum network density (0.005-0.024) is below the minimum tested density (0.05).
Table R4. CellOracle network sizes across 14 datasets.
| Dataset | # of retained edges (network sizes evaluated) |
|---|---|
| Adamson | 6,105 (5k, full) |
| Drager | 7,720 (5k, full) |
| Endothelial | 2,446 (full) |
| GWCD4i | 10,414 (5k, 10k, full) |
| Leng2021 | 12,634 (5k, 10k, full) |
| MOLM-13 | 7,453 (5k, full) |
| Norman | 11,463 (5k, 10k, full) |
| Replogle K562 | 11,344 (5k, 10k, full) |
| Replogle RPE1 | 10,986 (5k, 10k, full) |
| SARS-CoV-2 | 9,152 (5k, full) |
| Schmidt2022 CRISPRa | 8,635 (5k, full) |
| Tian2020 CRISPRi | 3,540 (full) |
| Tian2020 CRISPRa | 6,452 (5k, full) |
| WholeBrain | 10,674 (5k, 10k, full) |
We will clarify these evaluation settings in the revised manuscript and report the per-dataset network sizes. We acknowledge that differences in attainable network sizes limit direct comparisons on density-sensitive metrics.
Response to Reviewer Q5DD (4/4)
Response to Reviewer Q5DD (4/4)
Summary
We thank the reviewer for highlighting these important evaluation concerns. Our additional analyses demonstrate the robustness of GO-based rankings, highlight the distinct information captured by different evaluation strategies, and clarify the effects of filtering, checkpoint selection, and cell sampling.
We believe these analyses substantially strengthen the benchmark's methodological rigor and transparency. We will incorporate these additional analyses and clarifications into the revised manuscript and would appreciate the reviewer's feedback on whether these results adequately address the main concerns.
We hope these clarifications and corresponding manuscript revisions will encourage the reviewer to reconsider their assessment.
Response to Reviewer Q5DD (1/4)
Response to Reviewer Q5DD (1/4)
We sincerely thank the reviewer for the detailed and constructive feedback. We conducted additional experiments and sensitivity analyses to address the concerns about evaluation fairness, robustness, and comparability.
Question 1
Question 1: “Ranking metrics use different evaluation sets. GO AUROC/AUPRC and DE AUROC are computed only over each method's selected edges (Appendices F.2/G.3, pp. 21-22), measuring within-network ordering rather than recovery over a shared candidate space. Do the rankings persist when AUROC/AUPRC are evaluated on a shared annotated/testable candidate set, with unavailable scores handled explicitly?”
We thank the reviewer for highlighting this important evaluation concern. Below, we first explain why we chose to evaluate different sets of edges per method; then, following the reviewer's suggestion, we present new results on the same shared candidate set across methods, with interpretations.
Rationale of our choice and prior work. Evaluating method-specific edge sets has precedent in GRN benchmarking. For example, both Pan et al. (Nature Communications, 2025) computed AUROC and AUPRC on top-ranked edges by each method, and Skok Gibbs et al. (Genome Biology, 2024) used overlap AUPRC within edge sets identified by each method. Similarly, our original evaluation intentionally measures ranking quality within selected edges, which is designed to complement edge selection evaluation based on OR.
Additional shared-candidate evaluation. We agree with the reviewer that evaluating all methods on the same shared set will offer additional insights into method comparison. We evaluated all methods on identical candidate sets: gene pairs with GO annotations for both genes in the functional evaluation, and DE-testable gene pairs with at least one regulator in the perturbation evaluation. Unselected or uncovered edges received the lowest tied score. As both selected and unselected edges are ranked, this new evaluation will assess both edge recovery and within-network ranking.
Findings. GO rankings remain highly consistent, supporting the robustness of our functional findings. In contrast, shared-candidate DE AUROC rankings differ substantially from selected-edge rankings but are highly correlated with DE OR (Spearman ρ = 0.94) (Table R1). This highlights two important points: (1) within-network ranking and shared-candidate recovery may capture distinct aspects of network inference, consistent with our discussion on page 8 (lines 424-430); and (2) shared-candidate AUROC may provide partly redundant information with the edge-selection metric DE OR, further motivating our original selected-edge evaluation as a complementary measure of ranking quality.
Table R1. Ranking consistency between selected-edge and shared-candidate evaluations.
We will include both evaluations and clarify their complementary interpretations in the revised manuscript.
Reference
Skok Gibbs, C., Mahmood, O., Bonneau, R. et al. PMF-GRN: a variational inference approach to single-cell gene regulatory network inference using probabilistic matrix factorization. Genome Biol 25, 88 (2024). https://doi.org/10.1186/s13059-024-03226-6
Pan, Q., Ding, L., Hladyshau, S. et al. scMINER: a mutual information-based framework for clustering and hidden driver inference from single-cell transcriptomics data. Nat Commun 16, 4305 (2025). https://doi.org/10.1038/s41467-025-59620-6