acceptodds
Under review as a conference paper at ICLR 2027

GNEval: Benchmarking of context-specific foundation models for gene network inference

Abstract

Gene networks encode regulatory and functional relationships among genes. Single-cell foundation models, pretrained on large and diverse datasets, provide a range of gene-gene association signals that offer a promising basis for gene network inference. However, it remains unclear which gene network properties these signals capture and how they compare with domain-specific methods. To address this gap, we introduce GNEval, a unified benchmark for evaluating pretrained model derived gene associations for network inference. We benchmark 21 gene representations and association scores from 11 models, including static gene embeddings (S), contextual embeddings (C), and gene-gene attention scores (A). GNEval spans 14 Perturb-seq datasets across diverse cell systems, perturbation types and data sizes. Inferred gene networks are assessed based on their consistency with functional and perturbational evidence. The results reveal substantial heterogeneity across evaluation tasks. Rankings based on Gene Ontology and Perturb-seq evidence were nearly uncorrelated (Spearman ). Context-independent representations showed a substantial advantage in recovering functional relationships (), whereas context-dependent gene associations improved recovery of perturbation-associated differential expression relationships by . The domain-specific method achieves relatively strong performance on both evidence types, offering a better balance across the two evaluations than most foundation-model-derived networks. Overall, gene representation and association scores show task-specific strengths, highlighting the need for task-aware selection and integration with domain-specific methods.

Primary area:
datasets and benchmarks
Keywords:
Representation Learning, Foundation Models, Gene Network Inference, Single-Cell Genomics, Benchmarking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Q5DDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GNEval benchmarks gene-network inference using 21 gene representations or association scores from 11 models across 14 Perturb-seq datasets. It compares static embeddings, contextual embeddings, attention-derived scores, and domain-specific methods using GO-based functional relationships and perturbation-associated differential-expression evidence. The authors report markedly different rankings under the two evidence sources. The context-independent group performs better on the GO evaluations, whereas the pooled context-dependent group, comprising contextual embeddings and attention-derived scores, achieves higher regulator-level perturbational enrichment; DE AUROC instead favors the context-independent group. The paper also provides a harmonized preprocessing pipeline, per-dataset results, sensitivity analyses, and computational-cost measurements.

Critical strengths
  1. Comparing functional and perturbational evidence across 14 datasets provides a useful perspective on the task-dependent strengths of gene representations.
  2. Detailed preprocessing documentation, per-dataset results, sensitivity analyses, and computational-cost measurements improve the benchmark's transparency and practical utility.
Critical weaknesses
  1. Ranking metrics use different evaluation sets. GO AUROC/AUPRC and DE AUROC are computed only over each method's selected edges (Appendices F.2/G.3, pp. 21-22), measuring within-network ordering rather than recovery over a shared candidate space.
  2. Filtering may affect aggregate comparisons. Appendix G.4 excludes combinations with fewer than ten DE-supported edges from AUROC and odds-ratio summaries, while Appendix I acknowledges non-random missingness; method medians can therefore reflect different dataset subsets, without a reported common-dataset sensitivity analysis.
  3. The contribution of context is not isolated. The static and contextual groups contain different model/readout mixtures, and Geneformer uses V2-316M for static embeddings versus V2-104M for contextual embeddings and attention (Table 5); these differences limit attribution of the group-level results to target-dataset context.
  4. Baseline inputs are not matched. Contextual methods use 1,000 cells from the full Perturb-seq population (Section 2.3), whereas CS-CORE uses all non-targeting control cells (Appendix D.3), mixing differences in cell composition and sample size with differences between methods.
Questions for authors
  1. Do the rankings persist when AUROC/AUPRC are evaluated on a shared annotated/testable candidate set, with unavailable scores handled explicitly?
  2. How do rankings change when low-hit cases are retained where metrics remain defined and methods are compared on the same datasets?
  3. Does the static-versus-contextual difference persist with identical checkpoints and common represented genes, analyzing attention separately?
  4. How do contextual methods and CS-CORE compare when both use the same number of non-targeting control cells?
  5. What edge counts were used for CellOracle in each dataset, and was it included in the network-size and fixed-density analyses?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Q5DD (1/4)

Response to Reviewer Q5DD (1/4)

We sincerely thank the reviewer for the detailed and constructive feedback. We conducted additional experiments and sensitivity analyses to address the concerns about evaluation fairness, robustness, and comparability.

Question 1

Question 1: “Ranking metrics use different evaluation sets. GO AUROC/AUPRC and DE AUROC are computed only over each method's selected edges (Appendices F.2/G.3, pp. 21-22), measuring within-network ordering rather than recovery over a shared candidate space. Do the rankings persist when AUROC/AUPRC are evaluated on a shared annotated/testable candidate set, with unavailable scores handled explicitly?”

We thank the reviewer for highlighting this important evaluation concern. Below, we first explain why we chose to evaluate different sets of edges per method; then, following the reviewer's suggestion, we present new results on the same shared candidate set across methods, with interpretations.

Rationale of our choice and prior work. Evaluating method-specific edge sets has precedent in GRN benchmarking. For example, both Pan et al. (Nature Communications, 2025) computed AUROC and AUPRC on top-ranked edges by each method, and Skok Gibbs et al. (Genome Biology, 2024) used overlap AUPRC within edge sets identified by each method. Similarly, our original evaluation intentionally measures ranking quality within selected edges, which is designed to complement edge selection evaluation based on OR.

Additional shared-candidate evaluation. We agree with the reviewer that evaluating all methods on the same shared set will offer additional insights into method comparison. We evaluated all methods on identical candidate sets: gene pairs with GO annotations for both genes in the functional evaluation, and DE-testable gene pairs with at least one regulator in the perturbation evaluation. Unselected or uncovered edges received the lowest tied score. As both selected and unselected edges are ranked, this new evaluation will assess both edge recovery and within-network ranking.

Findings. GO rankings remain highly consistent, supporting the robustness of our functional findings. In contrast, shared-candidate DE AUROC rankings differ substantially from selected-edge rankings but are highly correlated with DE OR (Spearman ρ = 0.94) (Table R1). This highlights two important points: (1) within-network ranking and shared-candidate recovery may capture distinct aspects of network inference, consistent with our discussion on page 8 (lines 424-430); and (2) shared-candidate AUROC may provide partly redundant information with the edge-selection metric DE OR, further motivating our original selected-edge evaluation as a complementary measure of ranking quality.

Table R1. Ranking consistency between selected-edge and shared-candidate evaluations.

MetricSpearman ρKey observation
GO AUROC0.94GenePT remains #1
GO AUPRC0.97GenePT remains #1
DE AUROC0.28Highly correlated with DE OR (Spearman ρ = 0.94), indicating that both DE OR and DE AUROC are capturing edge selection performance.

We will include both evaluations and clarify their complementary interpretations in the revised manuscript.

Reference

Skok Gibbs, C., Mahmood, O., Bonneau, R. et al. PMF-GRN: a variational inference approach to single-cell gene regulatory network inference using probabilistic matrix factorization. Genome Biol 25, 88 (2024). https://doi.org/10.1186/s13059-024-03226-6

Pan, Q., Ding, L., Hladyshau, S. et al. scMINER: a mutual information-based framework for clustering and hidden driver inference from single-cell transcriptomics data. Nat Commun 16, 4305 (2025). https://doi.org/10.1038/s41467-025-59620-6

AuthorsOpenReview ↗

Response to Reviewer Q5DD (2/4)

Response to Reviewer Q5DD (2/4)

Question 2

Question 2: “Filtering may affect aggregate comparisons. Appendix G.4 excludes combinations with fewer than ten DE-supported edges from AUROC and odds-ratio summaries, while Appendix I acknowledges non-random missingness; method medians can therefore reflect different dataset subsets, without a reported common-dataset sensitivity analysis. How do rankings change when low-hit cases are retained where metrics remain defined and methods are compared on the same datasets?”

We thank the reviewer for highlighting the potential selection bias introduced by excluding low-hit cases in perturb-seq-based evaluation. We agree that this is an important concern. The original minimum threshold of 10 DE-supported edges was intended to reduce noise from sparse positive labels. To address this concern, we (1) lowered the minimum number of DE-supported edges from 10 to 1, retaining all cases with defined metrics; (2) included zero-hit cases by assigning DE OR = 0 and DE AUROC = 0.5, with the latter serving as a neutral ranking convention because AUROC is undefined without positive edges; (3) reduced the minimum number of testable regulators for enrichment evaluation from 10 to 1; and (4) extended the analysis to all 14 datasets.

The rankings remain robust after we include 14 datasets with a perturb-seq-based rank correlation of 0.913 across all 21 methods. Importantly, our main conclusion remained unchanged: context-dependent associations showed stronger recovery of perturbation evidence.

We acknowledge that assigning DE AUROC = 0.5 may favor zero-hit methods over those with slightly below-chance performance. For example, scFoundation_C moves from 21st to 9th in DE AUROC and from 21st to 17th in perturb-seq-based rank. We therefore interpret these rankings with caution and emphasize the stability of the overall conclusions rather than exact method positions.

We will include both evaluations and clarify their complementary interpretations in the revised manuscript.

Question 3

Question 3: “The contribution of context is not isolated. The static and contextual groups contain different model/readout mixtures, and Geneformer uses V2-316M for static embeddings versus V2-104M for contextual embeddings and attention (Table 5); these differences limit attribution of the group-level results to target-dataset context. Does the static-versus-contextual difference persist with identical checkpoints and common represented genes, analyzing attention separately?”

We thank the reviewer for identifying the checkpoint mismatch. To address this, we evaluated static Geneformer using V2-104M, matching the checkpoint used for contextual embeddings and attention-derived scores. We directly compared all three representations under the same evaluation protocol, retaining V2-316M static embeddings as a reference. Paired Wilcoxon tests were performed on dataset-level differences, with BH correction across six metrics.

We made two key observations from Table R2:

1. Static representations show broadly consistent performance across checkpoints, as shown by columns (1) and (2).

2. The static-versus-contextual differences largely persist under identical checkpoints, as shown by comparing column (2) with columns (3) and (4). This pattern holds for all three GO metrics (q<0.01), DE OR, and significant-overlap rate. Although the DE AUROC difference reverses direction after checkpoint matching, it remains statistically nonsignificant under both checkpoints.

Table R2. Geneformer representation comparison with matched checkpoints.

Metric(1) Static(2) Static(3) Contextual(4) Attention
CheckpointV2-316MV2-104MV2-104MV2-104M
GO Fisher OR5.65 (4.0)5.47 (4.1)2.77 (8.7)0.90 (16.2)
GO AUROC0.678 (5.6)0.699 (5.4)0.632 (10.7)0.532 (14.6)
GO AUPRC0.081 (5.0)0.077 (4.6)0.023 (9.2)0.003 (15.4)
DE OR1.44 (12.7)1.36 (14.9)1.73 (10.6)3.83 (3.9)
DE AUROC0.519 (7.4)0.484 (12.9)0.492 (9.2)0.496 (9.3)
Sig. overlap0.006 (13.2)0.000 (14.2)0.022 (9.7)0.056 (4.2)

Values are median performance across datasets, with mean within-dataset ranks among 21 methods in parentheses (lower ranks are better). Results use the top-K evaluation (5k/10k/20k) and GO Jaccard > 0.3.

These results confirm that the GO findings are robust to checkpoint selection, while some perturbation-based comparisons are checkpoint-sensitive. We will use matched checkpoints in the revised manuscript and avoid attributing group-level differences solely to contextualization.

AuthorsOpenReview ↗

Response to Reviewer Q5DD (3/4)

Response to Reviewer Q5DD (3/4)

Question 4

Question 4: “Baseline inputs are not matched. Contextual methods use 1,000 cells from the full Perturb-seq population (Section 2.3), whereas CS-CORE uses all non-targeting control cells (Appendix D.3), mixing differences in cell composition and sample size with differences between methods. How do contextual methods and CS-CORE compare when both use the same number of non-targeting control cells?”

We thank the reviewer for highlighting the differences in cell sampling. We agree that using the same cells will enable a fairer comparison of contextual methods and CS-CORE. We reran all 11 contextual/attention methods and CS-CORE using identical non-targeting control cells, including all available controls. The two largest datasets GWCD4i and WholeBrain were capped at 12,000 cells due to their large control pools (309,554 and 195,132 cells, respectively), with the cap exceeding the largest uncapped dataset (Norman, 11,483 cells).

The rankings remain highly consistent for GO-based metrics (ρ = 0.972-0.993) and reasonably stable for perturbation-based metrics (ρ = 0.818-0.909) (Table R3). These results indicate that method rankings are largely robust to control-cell sample size, although some perturbation-based rankings are more sensitive. Importantly, all contextual methods and CS-CORE now use matched control-cell inputs, addressing the original concern about baseline comparability.

Table R3. Ranking consistency between 1,000 and all available control cells (12 methods).

MetricSpearman ρMax. rank changeMethod with largest change
GO OR0.9722GeneCompass_A
GO AUROC0.9861Geneformer_A
GO AUPRC0.9931GeneCompass_A
DE OR0.9093scPRINT_A
DE AUROC0.8184CS-CORE
Sig. overlap0.8604scPRINT_A

We will update the manuscript to use matched control-cell inputs and clarify the sampling protocol.

Question 5

Question 5: "What edge counts were used for CellOracle in each dataset, and was it included in the network-size and fixed-density analyses?"

We thank the reviewer for this question. Unlike other methods, CellOracle produces sparse networks restricted to TF-target pairs supported by its base GRN inferred from external data. Following the recommended filtering procedure (p < 0.001), the resulting networks contain 2,446-12,634 edges across datasets (Table R4).

Network-size (top-K) analysis: CellOracle is evaluated at all available network sizes (5k, 10k, and 20k edges), plus its full network, and is excluded from network sizes exceeding its available edge count.

Fixed-density and rank-stability analyses: CellOracle is excluded because its maximum network density (0.005-0.024) is below the minimum tested density (0.05).

Table R4. CellOracle network sizes across 14 datasets.

Dataset# of retained edges (network sizes evaluated)
Adamson6,105 (5k, full)
Drager7,720 (5k, full)
Endothelial2,446 (full)
GWCD4i10,414 (5k, 10k, full)
Leng202112,634 (5k, 10k, full)
MOLM-137,453 (5k, full)
Norman11,463 (5k, 10k, full)
Replogle K56211,344 (5k, 10k, full)
Replogle RPE110,986 (5k, 10k, full)
SARS-CoV-29,152 (5k, full)
Schmidt2022 CRISPRa8,635 (5k, full)
Tian2020 CRISPRi3,540 (full)
Tian2020 CRISPRa6,452 (5k, full)
WholeBrain10,674 (5k, 10k, full)

We will clarify these evaluation settings in the revised manuscript and report the per-dataset network sizes. We acknowledge that differences in attainable network sizes limit direct comparisons on density-sensitive metrics.

AuthorsOpenReview ↗

Response to Reviewer Q5DD (4/4)

Response to Reviewer Q5DD (4/4)

Summary

We thank the reviewer for highlighting these important evaluation concerns. Our additional analyses demonstrate the robustness of GO-based rankings, highlight the distinct information captured by different evaluation strategies, and clarify the effects of filtering, checkpoint selection, and cell sampling.

We believe these analyses substantially strengthen the benchmark's methodological rigor and transparency. We will incorporate these additional analyses and clarifications into the revised manuscript and would appreciate the reviewer's feedback on whether these results adequately address the main concerns.

We hope these clarifications and corresponding manuscript revisions will encourage the reviewer to reconsider their assessment.

Reviewer_jqdVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presented GNEval, a systematic evaluation benchmark of the utility of context-specific gene embeddings or attention-derived association scores from scFMs for gene network inference. The authors develop a standardized pipeline for perturbation dataset processing, apply it to 14 publicly available datasets and run benchmark evaluations on 11 models. A reproducible implementation for benchmarking is provided under an anonymized repo.

Critical strengths

The paper is clearly presented, it is easy to follow and provides extensive evaluation across a wide range of models (11) and perturbation datasets (14).

Critical weaknesses

Contribution and value; The overall utility of the benchmark is unclear, namely, as presented does not seem to add much to the existing literature concerning scFMs performance. While highlighting the inclusion of context-dependent representations it again recovers the task sensitivity of single-cell representation models. Clearly this is not to blame the benchmark, as this is just a mirror of model performance. However, the critique is that as a reader I find that it did not leave me with new understandings nor provide a novel interesting evaluation framework (uses standard metrics).

Questions for authors

Can the authors elaborate what is the practical utility of the benchmark as they see it and the contribution of the work ? it will be valuable to address both both:
1.a the understandings that can be drawn for the single-cell modeling community;
1.b the relevance for the broader ML community and fit to ICLR.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jqdV (1/2)

Response to Reviewer jqdV (1/2)

Question 1: the understandings that can be drawn for the single-cell modeling community

Thank you for this question, which helped us sharpen the main message of the paper.

We believe the main contribution of this work to the single-cell modeling community is not simply another benchmark of foundation models, but an evaluation of a new approach to inferring gene networks from single-cell data. Most, if not all, single-cell gene network inference methods use only the dataset at hand, which can be very limited in sample size, for example in a collaborative study. This ignores the vast amount of accumulated evidence on gene-gene relationships that could be learned from the millions of cells in single-cell atlases.

We hypothesize that single-cell foundation models capture such relationships from existing single-cell data, and can therefore be highly valuable for learning gene networks in small-sample settings, such as a new dataset collected by a clinical collaborator for a specific question of interest. This is the motivation behind our study: to understand whether single-cell foundation models can serve as a tool to summarize the gene relationships learned from millions of cells and transfer them to assist network inference on a new dataset. Answering this requires first establishing how well these models perform on new data and how they compare with state-of-the-art domain-specific tools, which is what our benchmark provides.

Our results offer three takeaways: (1) no single foundation-model-derived signal is best, as performance depends strongly on the type of reference evidence; (2) static embeddings better capture functional relationships, whereas context-dependent signals better capture perturbation effects, which guides the choice of signal for a given goal; and (3) foundation models complement rather than replace domain-specific methods, which motivates integrating the two. Developing such integration methods is the natural next step of this research.

To make this clearer, we have revised the introduction (section 1) and discussion (section 4) to state this motivation and these takeaways explicitly.

AuthorsOpenReview ↗

Response to Reviewer jqdV (2/2)

Response to Reviewer jqdV (2/2)

Question 2: the relevance for the broader ML community and fit to ICLR.

Thank you for this constructive question. We believe the problem studied here is relevant well beyond single-cell biology.

At its core, our work addresses a general problem: inferring the dependency structure among features from a small dataset, using a foundation model pretrained on large-scale data as a transferable source of prior knowledge. This problem arises in many domains. In neuroimaging, functional connectivity networks among brain regions are typically estimated from clinical cohorts of tens of subjects, while foundation models for brain activity are now pretrained on thousands of recordings (e.g., BrainLM; Caro et al., ICLR 2024). In multivariate time series, dependency graphs among sensors, assets, or climate variables are often inferred from short records, while time-series foundation models are pretrained on massive corpora (e.g., Chronos, Ansari et al., TMLR 2024; TimesFM, Das et al., ICML 2024). In each case, the same question arises: can the relationship between features encoded in a foundation model improve network inference on a small new dataset?

Answering this requires addressing questions that are fundamentally about the foundation models: what relationship do foundation models capture, where is it captured (static embeddings, contextual embeddings, or attention), and how does it compare with classical statistical or computational methods applied to the target data alone? These are questions of representation analysis, in the spirit of prior ICLR work showing that attention maps in protein language models recover residue contacts (Rao et al., ICLR 2021). Our finding that different model embeddings capture different types of relationships, and that conclusions depend strongly on the reference evidence used, also offers a general lesson for evaluating learned representations.

Single-cell biology is a particularly suitable testbed for these questions, because Perturb-seq provides large-scale interventional evidence for validating inferred networks, and Gene Ontology databse curate existing literature on gene-gene relationship, which are rarely available in other domains. We also note that the ICLR call for papers explicitly includes applications to physical sciences, and datasets and benchmarks among its subject areas.

To clarify the broader relevance of our work, we have revised the Introduction (Section 1) and Discussion (Section 4) to explicitly highlight its implications beyond single-cell biology.

References

Ortega Caro, J. et al. BrainLM: A foundation model for brain activity recordings. International Conference on Learning Representations (ICLR), 565–576 (2024). https://proceedings.iclr.cc/paper_files/paper/2024/file/029ce70401321de3808b3ac39e1ab167-Paper-Conference.pdf

Ansari, A. F. et al. Chronos: Learning the language of time series. Transactions on Machine Learning Research (TMLR) (2024). https://openreview.net/forum?id=gerNCVqqtR

Das, A. et al. A decoder-only foundation model for time-series forecasting. Proceedings of Machine Learning Research 235, 10148–10167 (2024). https://proceedings.mlr.press/v235/das24c.html

Rao, R. et al. Transformer protein language models are unsupervised structure learners. International Conference on Learning Representations (ICLR) (2021). https://openreview.net/forum?id=fylclEqgvgd

Reviewer_jqdVOpenReview ↗

Thank you

Thank you for the clarifications and excuse the late response -- unfortunately my contribution concerns remain and thus i will keep my score.

Thank you again for the engagement.

Reviewer_z8kGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces GNEval, a benchmark for evaluating the ability of pretrained gene representations to recover biologically meaningful gene networks. The framework evaluates 21 representations and association scores from 11 models across 14 Perturb-seq datasets, considering static embeddings, contextualized embeddings, and attention-derived gene associations. Network quality is assessed using complementary functional (Gene Ontology) and perturbational (differential expression) evidence. The experiments reveal that context-independent representations generally perform better in recovering functional associations, whereas context-dependent representations show advantages in recovering perturbation-associated gene relationships under certain metrics. The study further demonstrates that rankings under functional and perturbational evaluations are nearly uncorrelated, highlighting the importance of task-specific evaluation.

Critical strengths
  1. Comprehensive benchmarking with complementary biological evaluation criteria.

The paper systematically compares diverse foundation-model representations and domain-specific approaches across 14 Perturb-seq datasets. Importantly, incorporating both functional and perturbational evidence provides a more comprehensive assessment than relying solely on curated functional annotations. This contributes a useful benchmarking resource and offers practical guidance for selecting gene representations.

  1. Important empirical findings on the limitations of foundation-model representations.

The finding that functional and perturbational network evaluations yield substantially different model rankings is particularly valuable. Furthermore, the observation that conventional or domain-specific methods remain competitive with foundation models challenges the assumption that large-scale pretraining necessarily produces superior gene network representations. These findings provide meaningful insights for future model development.

Critical weaknesses
  1. Limited validity of the evaluation references for assessing regulatory network inference.

The benchmark relies on GO-based functional associations and perturbation-induced differential expression as reference evidence. However, functional similarity does not necessarily imply direct regulation, and differential expression following perturbation can reflect indirect downstream effects rather than direct regulatory relationships. Furthermore, inferred networks are evaluated using symmetric association scores, without assessing regulatory directionality. Consequently, the benchmark primarily evaluates biological associations rather than the accuracy of inferred regulatory mechanisms, limiting the strength of its conclusions regarding gene network inference.

  1. Potential confounding factors in the comparison of context-dependent and context-independent representations.

Context-dependent representations are extracted from subsampled cells in the Perturb-seq datasets, potentially including cells whose perturbation responses contribute to the evaluation evidence. Although perturbation labels are not provided to the models, this raises questions about whether the reported advantages reflect generalizable regulatory knowledge or associations specific to the evaluated perturbation data. Additionally, restricting evaluation to approximately 1,000 highly variable genes limits the applicability of the findings to broader regulatory networks. Evaluations using control-only expression profiles and larger gene sets would strengthen the central conclusions.

Questions for authors

See the Weakness section

2 comments
AuthorsOpenReview ↗

Response to Reviewer z8kG (1/2)

Response to Reviewer z8kG (1/2)

We thank the reviewer for the constructive feedback. We appreciate the concerns regarding biological interpretation and evaluation robustness, and address them below.

Question 1: Limited validity of the evaluation references for assessing regulatory network inference.

The benchmark relies on GO-based functional associations and perturbation-induced differential expression as reference evidence. [...] Consequently, the benchmark primarily evaluates biological associations rather than the accuracy of inferred regulatory mechanisms, limiting the strength of its conclusions regarding gene network inference.

We fully agree that this work evaluates association networks, commonly known as co-expression networks, rather than direct or directed regulatory relationships, and we should have made this clearer in the introduction. However, we respectfully argue that co-expression networks are an important class of gene networks in their own right. They emerged with the advent of high-throughput transcriptomic profiling (Stuart et al., Science, 2003) and have since become an indispensable tool for understanding how genes coordinate to perform biological functions and how dysregulation of biological pathways may underlie disease. For example, weighted gene co-expression network analysis (WGCNA; Langfelder and Horvath, BMC Bioinformatics, 2008) has received close to 30,000 citations and has been widely used across tissues and disease conditions, with successes in neuroscience (Geschwind and Konopka, Nature, 2009; Gandal et al., Science, 2018), cardiometabolic disease (Chen et al., Nature, 2008), and cancer (Segal et al., Nature Genetics, 2004). In the single-cell era, tools for cell-type-specific co-expression network inference have also emerged, including CS-CORE (Su et al., Nature Communications, 2023), which we evaluate in our work. Our results therefore have important implications for gene network inference, in particular for how trained foundation models can be leveraged to assist network inference in newly collected single-cell samples.

Regarding symmetric scores and directionality: because co-expression is an inherently symmetric relationship between two genes, we believe symmetric association scores are the appropriate way to evaluate the networks studied here. Likewise, GO-based functional associations and perturbation-induced differential expression are well suited as references, as both provide evidence that two genes are functionally related, which is precisely what a co-expression network aims to capture. We agree with the reviewer that the lack of directionality is a fundamental limitation of co-expression networks. Regulatory direction cannot be identified from observational expression data without additional assumptions or data, and is better captured by directed regulatory networks. Evaluating whether foundation models can support the inference of such networks is an important direction, which we leave for future work.

We thank the reviewer for this valuable comment, which prompted us to substantially clarify the scope and framing of the paper. In response, we have made the following revisions: (1) we rewrote the introduction (Section 1) to state explicitly that our benchmark evaluates co-expression networks rather than direct or directed regulatory relationships, and to motivate the importance of this class of networks with the literature cited above; (2) we added a paragraph to the discussion (Section 4) that acknowledges the lack of directionality as a fundamental limitation and outlines the evaluation of directed network inference as future work. We hope these revisions address the reviewer’s concern.

References:

Stuart, J. M. et al. A gene-coexpression network for global discovery of conserved genetic modules. Science 302, 249–255 (2003). https://doi.org/10.1126/science.1087447

Langfelder, P. & Horvath, S. WGCNA: an R package for weighted correlation network analysis. BMC Bioinformatics 9, 559 (2008). https://doi.org/10.1186/1471-2105-9-559

Geschwind, D. H. & Konopka, G. Neuroscience in the era of functional genomics and systems biology. Nature 461, 908–915 (2009). https://doi.org/10.1038/nature08537

Gandal, M. J. et al. Shared molecular neuropathology across major psychiatric disorders parallels polygenic overlap. Science 359, 693–697 (2018). https://doi.org/10.1126/science.aad6469

Chen, Y. et al. Variations in DNA elucidate molecular networks that cause disease. Nature 452, 429–435 (2008). https://doi.org/10.1038/nature06757

Segal, E. et al. A module map showing conditional activity of expression modules in cancer. Nat. Genet. 36, 1090–1098 (2004). https://doi.org/10.1038/ng1434

Su, C. et al. Cell-type-specific co-expression inference from single cell RNA-sequencing data. Nat. Commun. 14, 4846 (2023). https://doi.org/10.1038/s41467-023-40503-7

AuthorsOpenReview ↗

Response to Reviewer z8kG (2/2)

Response to Reviewer z8kG (2/2)

Question 2: Potential confounding factors in the comparison of context-dependent and context-independent representations.

Context-dependent representations are extracted from subsampled cells in the Perturb-seq datasets, potentially including cells whose perturbation responses contribute to the evaluation evidence. Although perturbation labels are not provided to the models, this raises questions about whether the reported advantages reflect generalizable regulatory knowledge or associations specific to the evaluated perturbation data. Additionally, restricting evaluation to approximately 1,000 highly variable genes limits the applicability of the findings to broader regulatory networks. Evaluations using control-only expression profiles and larger gene sets would strengthen the central conclusions.

We thank the reviewer for highlighting potential confounding factors.

Control-only evaluation. We thank the reviewer for highlighting the differences in cell sampling. We agree that using the same cells will enable a fairer comparison of contextual methods and CS-CORE. We reran all 11 contextual/attention methods and CS-CORE using identical non-targeting control cells, including all available controls. The two largest datasets GWCD4i and WholeBrain were capped at 12,000 cells due to their large control pools (309,554 and 195,132 cells, respectively), with the cap exceeding the largest uncapped dataset (Norman, 11,483 cells).

The rankings remain highly consistent for GO-based metrics (ρ = 0.972-0.993) and reasonably stable for perturbation-based metrics (ρ = 0.818-0.909) (Table R1). These results indicate that method rankings are largely robust to control-cell sample size, although some perturbation-based rankings are more sensitive. Importantly, all contextual methods and CS-CORE now use matched control-cell inputs, addressing the original concern about baseline comparability.

Table R1. Ranking consistency between 1,000 and all available control cells.

MetricSpearman ρMax. rank change
GO OR0.9722
GO AUROC0.9861
GO AUPRC0.9931
DE OR0.9093
DE AUROC0.8184
Significant overlap0.8604

We will include this matched-input analysis and clarify the sampling protocol in the revised manuscript.

Gene coverage. We used approximately 1,000 highly variable genes to establish a standardized comparison across models with different gene vocabularies and computational requirements. Similar gene-set sizes have also been used in recent single-cell perturbation and network modeling studies, including Lorch et al. (PNAS, 2026) and Jiang et al. (Cell, 2026).

Extending our evaluation to substantially larger gene sets would require rerunning Perturb-seq differential expression (DE) analysis and the entire benchmarking pipeline. Given the computational demands and limited revision timeframe, we are unable to complete these additional experiments at this stage. We acknowledge that this setting does not establish generalizability to larger gene networks and will clarify this limitation in the revised manuscript.

References:

Lorch, L. et al. Latent causal diffusions for single-cell perturbation modeling. Proc. Natl. Acad. Sci. U.S.A. 123, e2602630123 (2026). https://doi.org/10.1073/pnas.2602630123

Jiang, J. et al. D-SPIN constructs regulatory network models from scRNA-seq that reveal organizing principles of perturbation response. Cell 189, 4159–4184.e35 (2026). https://doi.org/10.1016/j.cell.2026.04.028

Summary

We appreciate the reviewer for highlighting these important issues. We believe the clarified evaluation scope and additional control-only experiments strengthen the biological interpretation and reliability of GNEval. We would welcome the reviewer's feedback on whether these analyses address the main soundness concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.