- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper examines how to organize existing biological knowledge into gene representations for predicting responses to unseen single-gene perturbations. The authors construct gene descriptions from public resources and encode them as text embeddings. They introduce an LLM-guided search procedure that iteratively refines a shared gene-description protocol based on validation performance in perturbation prediction.
The largest performance gains come from richer biological knowledge. More structured descriptions provide further improvements, while LLM-guided search adds a smaller but consistent benefit. The resulting text representations transfer across different predictors and models. The paper also uses transcriptome-wide log-fold changes from DESeq2 as prediction targets and introduces Continuous Directed Overlap (CDO), a metric that emphasizes the ranking of strongly responding genes and agreement in response direction.
Critical strengths
-
The paper uses a clear, controlled design to assess the effects of gene knowledge and description quality. It progressively compares gene names, NCBI/HGNC metadata, WikiCrow-enriched information, and canonical descriptions while keeping the encoder and predictor fixed.
-
The canonical rich-text representation consistently outperforms existing gene embeddings across datasets and metrics, even without protocol search. This supports its practical utility and suggests that the gains are not confined to the proposed CDO metric or a single search run.
Critical weaknesses
1. The representation-learning motivation is insufficiently connected to existing causal representation learning literature
The paper asks which gene representations best support prediction of unseen perturbations and emphasizes the generalization limits of foundation models and fixed gene embeddings. However, the related-work discussion focuses mainly on general-purpose or pretrained representations. It gives limited attention to causal representation learning methods developed for perturbation prediction, such as Discrepancy-VAE [1], sVAE+ [2], SAMS-VAE [3], and PerturbedVAE [4] . PerturbedVAE is particularly relevant in both motivation and problem setting, as it also examines whether foundation models learn perturbation-relevant representations that generalize to unseen perturbations.
The results demonstrate the usefulness of richer biological text embeddings, but the paper does not clearly explain how its approach differs from these causal representation learning methods in objective, assumptions, and generalization mechanism.
2. The empirical validation of CDO as a new evaluation metric remains limited
The paper presents CDO as a major methodological contribution for evaluating the ranking and directional consistency of strongly responding genes. The authors provide synthetic corruption tests, sensitivity analyses, and results across four Perturb-seq datasets, but validation remains largely confined to the paper’s own task and model settings.
Although the paper discusses limitations of Pearson correlation, MAE, top-K overlap, DEG recovery, and perturbation discrimination, it does not systematically compare CDO with existing perturbation evaluation metrics. It also remains unclear when these metrics disagree on model rankings, which failure modes each captures, and which better reflects biological relevance.
The results suggest that CDO has reasonable design properties, but do not yet demonstrate that it is more discriminative or reliable than existing metrics in practical perturbation prediction.
[1] Zhang J, Greenewald K, Squires C, et al. Identifiability guarantees for causal disentanglement from soft interventions[J]. Advances in Neural Information Processing Systems, 2023, 36: 50254-50292.
[2] Lopez R, Tagasovska N, Ra S, et al. Learning causal representations of single cells via sparse mechanism shift modeling[C]//Conference on Causal Learning and Reasoning. PMLR, 2023: 662-691.
[3] Bereket M, Karaletsos T. Modelling cellular perturbations with the sparse additive mechanism shift variational autoencoder[J]. Advances in Neural Information Processing Systems, 2023, 36: 1-12.
[4] Jiang W, Liu Y, Cai Y, et al. What Makes a Representation Good for Single-Cell Perturbation Prediction?[J]. arXiv preprint arXiv:2605.19343, 2026.
Questions for authors
Q1: Could the authors expand the discussion of causal or intervention-aware representation learning, including Discrepancy-VAE, sVAE+, SAMS-VAE, and PerturbedVAE? How does the proposed text-based approach differ from methods that explicitly learn perturbation-relevant or mechanism-aware representations in problem setting, learning objective, and generalization to unseen or OOD perturbations?
Q2: Could the authors systematically compare CDO with existing metrics, including DEG recovery, Jaccard/F1, and perturbation discrimination, and assess whether it adds useful information across more models, datasets, or real failure cases? When CDO ranks models differently from Pearson, MAE, or DEG-based metrics, can the authors show that its rankings better reflect biologically meaningful recovery of strong responses?
6 comments
Response to Reviewer rn5y (Part 2)
W2/Q2: Additional evaluation of CDO
We agree that CDO should be compared systematically with existing metrics. Our new evaluations show that improvements selected using CDO are accompanied by better DEG recovery, strong-response overlap, prediction accuracy and perturbation discrimination, even when correlation remains unchanged.
In the 218-perturbation K562 holdout (see response to FXRR), history-guided search improves CDO over canonical from 0.1546 to 0.1777 (paired Δ +0.0231 [0.0162, 0.0305]). MAE@100 decreases from 0.5095 to 0.4423, DEG AP increases from 0.1397 to 0.1645, and signed Jaccard from 0.1056 to 0.1376. PDS-L1 increases from 0.6291 to 0.7177 and PDS-cosine from 0.6445 to 0.7290. Corr@100 remains essentially unchanged.
We also evaluated GenePT, UniProt, ESM2, Geneformer, scGPT, GeneCompass and scFoundation on the 184-perturbation embedding-covered subset of the original 218 test perturbations, with 744 corresponding development perturbations. We reused protocols selected using the full 874-perturbation development set, without further optimisation, and refitted Ridge on the restricted development set. No test responses informed these new protocol searches or Ridge fits.
Prediction performance
| Gene representation | CDO ↑ | Corr@100 ↑ | MAE@100 ↓ |
|---|---|---|---|
| Mean response | 0.1105 [0.1041,0.1169] | 0.5955 [0.5625,0.6265] | 0.6152 [0.5747,0.6597] |
| GenePT | 0.1371 [0.1270,0.1473] | 0.6841 [0.6525,0.7140] | 0.5540 [0.5160,0.5946] |
| UniProt | 0.1266 [0.1184,0.1351] | 0.6734 [0.6461,0.6993] | 0.5720 [0.5320,0.6150] |
| ESM2 | 0.1206 [0.1132,0.1283] | 0.6564 [0.6276,0.6835] | 0.5852 [0.5462,0.6274] |
| Geneformer | 0.1183 [0.1106,0.1261] | 0.5917 [0.5566,0.6248] | 0.6045 [0.5642,0.6481] |
| GeneCompass | 0.1240 [0.1157,0.1325] | 0.6297 [0.5974,0.6600] | 0.5896 [0.5494,0.6339] |
| scGPT | 0.1292 [0.1200,0.1389] | 0.6364 [0.6052,0.6659] | 0.5809 [0.5398,0.6261] |
| scFoundation | 0.1104 [0.1041,0.1168] | 0.5965 [0.5639,0.6274] | 0.6150 [0.5744,0.6594] |
| Ind.-selected text | 0.1452 [0.1351,0.1556] | 0.7186 [0.6916,0.7441] | 0.5324 [0.4943,0.5734] |
| K562-selected text | 0.1703 [0.1558,0.1851] | 0.7232 [0.6915,0.7528] | 0.4618 [0.4270,0.4986] |
DEG recovery, strong-response overlap and perturbation discrimination
| Gene representation | DEG AP ↑ | Signed Jaccard ↑ | PDS-L1 ↑ | PDS-cosine ↑ |
|---|---|---|---|---|
| Mean response | 0.1014 [0.0848,0.1195] | 0.0743 [0.0700,0.0788] | 0.5000 [0.4580,0.5419] | 0.5000 [0.4584,0.5419] |
| GenePT | 0.1216 [0.1013,0.1441] | 0.0929 [0.0861,0.0995] | 0.5804 [0.5372,0.6221] | 0.5933 [0.5489,0.6363] |
| UniProt | 0.1160 [0.0965,0.1374] | 0.0869 [0.0806,0.0935] | 0.5617 [0.5182,0.6036] | 0.5744 [0.5317,0.6167] |
| ESM2 | 0.1108 [0.0922,0.1314] | 0.0813 [0.0760,0.0865] | 0.5415 [0.4985,0.5832] | 0.5515 [0.5089,0.5938] |
| Geneformer | 0.1054 [0.0878,0.1246] | 0.0855 [0.0794,0.0919] | 0.5140 [0.4724,0.5564] | 0.5217 [0.4789,0.5644] |
| GeneCompass | 0.1117 [0.0925,0.1328] | 0.0854 [0.0798,0.0912] | 0.5353 [0.4926,0.5781] | 0.5408 [0.4979,0.5840] |
| scGPT | 0.1155 [0.0957,0.1374] | 0.0901 [0.0833,0.0972] | 0.5447 [0.5018,0.5880] | 0.5581 [0.5143,0.6017] |
| scFoundation | 0.1015 [0.0849,0.1196] | 0.0743 [0.0700,0.0787] | 0.5002 [0.4583,0.5421] | 0.5004 [0.4588,0.5422] |
| Ind.-selected text | 0.1299 [0.1088,0.1531] | 0.0983 [0.0911,0.1054] | 0.6143 [0.5706,0.6568] | 0.6325 [0.5889,0.6747] |
| K562-selected text | 0.1530 [0.1295,0.1780] | 0.1308 [0.1174,0.1446] | 0.7144 [0.6736,0.7543] | 0.7293 [0.6859,0.7696] |
Values are means [95% perturbation-bootstrap confidence intervals] on the 184-perturbation test panel. DEG AP evaluates recovery of statistically significant DEGs; signed Jaccard evaluates overlap of responses exceeding a 1.5-fold threshold with matching direction; PDS-L1 and PDS-cosine measure perturbation discrimination using predicted and observed LFC vectors.
Optimised text achieves the best point estimates across all seven metrics. Relative to GenePT, the strongest existing embedding by CDO, DEG AP improves from 0.1216 to 0.1530, signed Jaccard from 0.0929 to 0.1308, and PDS-L1 from 0.5804 to 0.7144, with paired confidence intervals excluding zero.
Comparing independently selected and history-guided text, CDO increases from 0.1452 to 0.1703 (paired difference +0.0251 [0.0173, 0.0332]) and MAE@100 decreases from 0.5324 to 0.4618 (−0.0706 [−0.0851, −0.0570]). Corr@100 changes only from 0.7186 to 0.7232 (+0.0047 [−0.0083, 0.0176]), without a statistically resolved improvement.
Why does Corr@100 improve less? Corr@100 measures linear agreement within the 100 strongest observed responses, without assessing whether those genes rank highly among all predicted responses. It is also insensitive to positive rescaling and constant offsets. CDO instead evaluates recovery of strong responses in the correct direction and relative order, penalising displacement by other predicted responses. Improved ranking and effect-size accuracy can therefore coexist with unchanged correlation. The accompanying improvements in all metrics suggested by the reviewer show that the gains are not confined to CDO.
Response to Reviewer rn5y (Part 3)
Sensitivity to response thresholds. Because binary recovery metrics depend on how a strong response is defined, we repeated the signed Jaccard analysis at 1.25-, 1.5- and 2-fold expression-change thresholds, without changing the selected protocols or predictors. On the same 184-perturbation test panel:
| Fold-change threshold | Canonical | LLM-guided search | Paired improvement [95% CI] |
|---|---|---|---|
| 1.25-fold | 0.2333 | 0.2470 | +0.0137 [0.0043,0.0234] |
| 1.5-fold | 0.0989 | 0.1308 | +0.0319 [0.0227,0.0417] |
| 2-fold | 0.0595 | 0.0980 | +0.0385 [0.0243,0.0526] |
The improvement remains statistically resolved at every threshold and increases for stricter response definitions, supporting improved recovery of substantial transcriptional changes. This analysis assesses sensitivity of the binary overlap metric, rather than changing CDO's parameters.
Our claim is not that CDO replaces existing metrics or universally yields better model rankings. Rather, it provides a complementary rank- and direction-aware measure focused on strong transcriptional responses. The additional experiments show that improvements identified using CDO are accompanied by better prediction error, DEG recovery, strong-response overlap and perturbation discrimination.
We will incorporate the additional CDO evaluations and expanded discussion of causal representation learning into the revised manuscript. We thank the reviewer again for their constructive suggestions, which have helped us strengthen the empirical validation of CDO and clarify how our approach complements existing causal representation-learning methods.
Response to Reviewer rn5y (Part 1)
We thank the reviewer for their careful assessment and constructive suggestions. We particularly appreciate their recognition of our controlled representation comparisons and the utility of the resulting embeddings. We summarise our responses below:
W1/Q1 – Causal representation learning: We agree that Discrepancy-VAE, sVAE+, SAMS-VAE and PerturbedVAE are important related methods. While these approaches learn representations of cellular mechanisms and intervention effects, they do not, in their current formulations, support zero-shot prediction for genes never perturbed during training. Their generalisation capabilities instead include previously unseen combinations of known interventions. Our approach directly addresses unseen gene perturbations by constructing representations from external biological knowledge, without requiring experimental responses for those genes. We will expand the Related Work accordingly.
W2/Q2 – Validation of CDO: We evaluated DEG recovery, signed strong-response overlap, prediction error and perturbation discrimination on held-out K562 perturbations. On 218 test perturbations, CDO increases from 0.1546 to 0.1777, while MAE@100 decreases from 0.5095 to 0.4423 and DEG AP increases from 0.1397 to 0.1645, even though Corr@100 does not improve.
W2/Q2 – Broader metric comparisons: Across ten representations evaluated on the 184-perturbation embedding-covered subset of the same K562 holdout, optimised text achieves the best point estimates on all seven metrics. Improvements in signed Jaccard also persist at 1.25-, 1.5- and 2-fold response thresholds.
We provide further details and results below.
W1/Q1: Relationship to causal representation learning
We thank the reviewer for highlighting Discrepancy-VAE, sVAE+, SAMS-VAE and PerturbedVAE. We agree that these are important contributions to perturbation modelling and will expand our Related Work section accordingly.
These approaches investigate how cellular states and intervention effects can be represented or inferred from experimental observations. Discrepancy-VAE studies causal disentanglement and identifiability under soft interventions; sVAE+ models sparse mechanism shifts; SAMS-VAE models sparse, composable perturbation effects; and PerturbedVAE separates perturbation-invariant and responsive latent factors while incorporating causal structure. These methods offer important advantages in learning structured representations of cellular mechanisms and modelling intervention effects. Several, including SAMS-VAE and PerturbedVAE, also generalise to unseen combinations of known interventions.
However, these methods do not, in their current formulations, support zero-shot prediction of the response to perturbing an entirely unseen gene. SAMS-VAE and PerturbedVAE can generalise to unseen combinations of interventions, but only when their constituent effects have been learned during training. The authors of PerturbedVAE explicitly acknowledge this limitation, identifying additional gene-level information, including biological annotations or gene embeddings, as a potential route to unseen-gene prediction (Appendix B). sVAE+ also evaluates unseen interventions, but through transfer learning, fine-tuning intervention-specific parameters using expression data from the held-out perturbations. Thus, while these approaches offer valuable forms of generalisation, they cannot directly predict the effects of previously unperturbed genes from their biological identity alone.
Our work addresses this challenge by constructing representations directly from prior biological knowledge about the intervened gene. We optimise a shared, human-readable gene-description protocol using downstream prediction feedback, while keeping the description generator, encoder and predictor fixed. This enables predictions for genes with no experimental perturbation responses in training. Unlike the causal approaches above, however, we do not explicitly learn latent cellular mechanisms or claim causal identifiability.
We therefore see considerable potential in combining the strengths of both approaches. Our matched benchmarks, consistent with Cole et al. (2026), demonstrate the predictive value of knowledge-derived gene representations, although they do not directly compare against these causal models. Incorporating such representations, potentially enriched with interaction networks and pathway information, as priors within causal or intervention-aware models could provide a means of generalising to entirely unseen gene targets while retaining their ability to learn structured cellular mechanisms and compositional perturbation effects. We will expand the Related Work and Discussion to clarify this distinction and the opportunities for integration.