- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors proposed CombPert, a biological knowledge-guided LLM reasoning framework for predicting perturbation responses combinatorially. The work builds on LLM-enhanced methods, for example, Scouter, and argues that LLM-enhanced methods have limited generalisation power for genes partially seen or unseen in the training data. They instead propose to use prior knowledge through constructing a task-specific biological knowledge graph from curated databases and dataset-specific co-expression statistics. Gene similarity and conditional priors are bundled into a CoT prompt for prediction and reasoning (reasoning elicitation) and distillation onto smaller language models (reasoning synthesis). CombPert with GPT 5.4 outperforms LLM-only, GEARS, scLAMBDA, and Scouter on Norman et al. and Replogle et al. datasets for the top 20 DEGs in most settings.
Critical strengths
-
Combinatorial generalisation to unseen perturbations is an important challenge, and the question is well motivated by observation, and the writing is generally clear.
-
The paper explored the idea of whether having task-specific prior knowledge would help LLMs to predict combinatorial perturbation direction and response value accurately, with the added advantage of explicit reasoning.
-
The datasets and baselines are appropriately selected, with clear performance gains on the 20 DEGs subset of the Norman et al. dataset.
Critical weaknesses
-
The framework is mainly an assembly of retrieval from KG priors and prompting techniques (CoT, CoT distillation) applied to a new task, with most gains over GEARS on unseen perturbations coming from a strong LLM only (GPT 5.4).
-
CombPert is given the (pair, gene), and operates only on the DEGs in the main result section, unlike GEARS which works on the full gene set. This requires the model in deployment to predict the DEGs first.
-
The effect of reasoning synthesis is weak to marginal compared to reasoning elicitation. The student model retains the full retrieval context, and it is unclear if there is any gain coming from the CoT traces.
-
The results section does not report run-to-run variations with a non-zero temperature. This makes it difficult to separate the real gains from LLM stochasticity.
-
The paper argues that unseen perturbation performance is due to knowledge-guided reasoning, but the evidence does not isolate this. Open-weight models generally lose to GEARS on unseen perturbation pairs on MSE. The effect of the approach over GEARS could be due to a strong and generalisable LLM (GPT5.4) instead.
-
The method relies on the quality of curated prior knowledge, which might be biased or contain errors. This should be highlighted.
Questions for authors
-
Can the LLM reliably predict the DEGs before predicting the responses?
-
Do the fine-tuned student models still receive the prior knowledge paths/priors at inference?
-
If noise is purposefully introduced into the prior knowledge KG, would CombPert performance degrade?
-
What are the test set sizes for the seen/unseen perturbation settings? They appear to be relatively small. Then the LLM stochasticity could weigh more.
7 comments
Response to Reviewer vXVA (Part 2)
Concern 3: The effect of reasoning synthesis is weak to marginal compared to reasoning elicitation. The student model retains the full retrieval context, and it is unclear if there is any gain coming from the CoT traces.
We agree that the gains over Reasoning Elicitation are modest and vary by metric. On Norman, Reasoning Synthesis reduces overall MSE from 0.241 to 0.216 for Qwen3-8B and from 0.207 to 0.194 for Qwen3-32B. The students retain the retrieval context at inference because the objective is to improve prediction given that evidence. Appendix B.8 (Table 13) of the submitted manuscript also compares training data generated by the same GPT-5.4 teacher using either CombPert or few-shot prompting. Under matched fine-tuning and evaluation settings, training on CombPert-generated data reduces overall MSE from 0.562 to 0.216 for Qwen3-8B and from 0.394 to 0.194 for Qwen3-32B. This comparison supports the utility of CombPert-generated training data.
Besides, to isolate the contribution of CoT traces, we fine-tune Qwen3-8B on the same CombPert-generated examples from Norman. Answer-only SFT uses only the teacher’s DE value, while CoT SFT uses the same value and its reasoning trace. Both variants use the same initialization, fine-tuning settings, and retrieved evidence, and are evaluated on identical test queries.
| Training variant | Overall MSE ↓ | Overall PCC ↑ | Overall ACC ↑ |
|---|---|---|---|
| Answer-only SFT | 0.264 | 0.793 | 0.901 |
| CoT SFT (Reasoning Synthesis) | 0.216 | 0.832 | 0.928 |
Adding CoT trace supervision reduces overall MSE from 0.264 to 0.216. With the training examples, numerical targets, and retrieval context held fixed, this improvement quantifies the additional predictive value of the CoT traces.
Concern 4: The results section does not report run-to-run variations with a non-zero temperature. This makes it difficult to separate the real gains from LLM stochasticity.
Appendix B.5 and Tables 9–10 of the submitted manuscript already report the mean and standard error for MSE, PCC, and direction accuracy. These estimates summarize variation across test perturbation conditions at a decoding temperature of 0.6.
Concern 5: The paper argues that unseen perturbation performance is due to knowledge-guided reasoning, but the evidence does not isolate this. Open-weight models generally lose to GEARS on unseen perturbation pairs on MSE. The effect of the approach over GEARS could be due to a strong and generalisable LLM (GPT5.4) instead.
Thank you for raising this point. Table 3 of the submitted manuscript already reports a comparison with the same GPT-5.4 backbone and input evidence. In the w/o Biological Guidance variant, the LLM directly predicts DE from the assembled evidence after we remove the task-specific biological instructions. The comparison below measures the contribution of those instructions on Seen 0/2.
| Method | Norman MSE ↓ | Norman PCC ↑ | Norman ACC ↑ | Replogle MSE ↓ | Replogle PCC ↑ | Replogle ACC ↑ |
|---|---|---|---|---|---|---|
| GEARS | 0.254 | 0.800 | 0.882 | 0.061 | 0.583 | 0.500 |
| w/o Biological Guidance | 0.283 | 0.760 | 0.903 | 0.318 | 0.482 | 0.700 |
| CombPert | 0.232 | 0.796 | 0.917 | 0.176 | 0.588 | 0.790 |
With the backbone and evidence held fixed, adding biological guidance reduces MSE from 0.283 to 0.232 on Norman and from 0.318 to 0.176 on Replogle. On Norman, GEARS’s MSE of 0.254 lies between the two GPT-5.4 variants, showing that biological guidance contributes to CombPert’s lower MSE in this setting. On Replogle, the guidance also improves prediction, while GEARS retains a lower MSE. As discussed in our response to Concern 1, the framework also improves prediction with an open-weight backbone. On Norman Seen 0/2, training-free CombPert reduces Qwen3-32B’s MSE from 0.540 to 0.321 relative to few-shot prompting with the same backbone.
Together, these results support a contribution from task-specific biological guidance under a fixed backbone and show that performance relative to GEARS varies across datasets and backbones.
Concern 6: The method relies on the quality of curated prior knowledge, which might be biased or contain errors. This should be highlighted.
We agree that the quality of curated biological knowledge matters. Appendix D (Limitations) of the submitted manuscript explicitly notes that incomplete or noisy biological knowledge can affect evidence retrieval and prediction quality. Besides, Appendix B.4 illustrates this sensitivity on Norman. With the numerical prior and evidence from analogous perturbations held fixed, reversing a key path increases MSE from 0.159 to 0.256.
Response to Reviewer vXVA (Part 3)
Question 1: Can the LLM reliably predict the DEGs before predicting the responses?
CombPert can predict DE for each gene in the measured panel directly, without prior DEG screening. Appendix B.9 (Table 14) of the submitted manuscript reports this full-panel evaluation on Norman. As discussed in our response to Concern 2, the top-20 restriction applies to the primary evaluation subset.
Question 2: Do the fine-tuned student models still receive the prior knowledge paths/priors at inference?
Yes. The fine-tuned students receive the same types of evidence as in Reasoning Elicitation, including retrieved KG paths, evidence from analogous perturbations, and the conditional statistical prior. As discussed in our response to Concern 3, Reasoning Synthesis trains the student to make predictions given this evidence.
Question 3: If noise is purposefully introduced into the prior knowledge KG, would CombPert performance degrade?
Appendix B.4 (Figure 9) tests sensitivity to errors in the retrieved paths. On Norman, reversing a key path increases MSE from 0.159 to 0.256 while holding the numerical prior and evidence from analogous perturbations fixed. This experiment tests corruption of a retrieved path at inference. Our response to Concern 6 discusses the resulting limitation.
Question 4: What are the test set sizes for the seen/unseen perturbation settings? They appear to be relatively small. Then the LLM stochasticity could weigh more.
We evaluate CombPert on two established Perturb-seq datasets, Norman and Replogle. Tables 6 and 7 of the submitted manuscript report the test set sizes. Each double-perturbation condition contributes 20 target-gene queries to the main evaluation. We report both counts below.
| Dataset | Seen 2/2 conditions / queries | Seen 1/2 conditions / queries | Seen 0/2 conditions / queries |
|---|---|---|---|
| Norman | 14 / 280 | 48 / 960 | 17 / 340 |
| Replogle | 5 / 100 | 12 / 240 | 3 / 60 |
Individual perturbation conditions can have a greater influence on aggregate scores in smaller splits, particularly Replogle Seen 0/2. To quantify uncertainty in the reported mean performance, Appendix B.5 (Tables 9–10) reports standard errors for MSE, PCC, and direction accuracy across test perturbation conditions. As discussed in our response to Concern 4, these estimates describe uncertainty across conditions.
Response to Reviewer vXVA (Part 1)
Dear Reviewer vXVA:
We sincerely thank you for your review and invaluable feedback. To better explain our work, we will make the following clarifications about your concerns:
We thank the reviewer for raising these points. We address them in two parts.
Task-specific biological guidance. CombPert organizes mechanistic paths, evidence from analogous perturbations, and statistical priors into a scaffold for combinatorial perturbation prediction. The guidance directs the LLM to assess the individual effects of the two perturbations, use cross-interaction and convergence paths to assess their joint effect, and estimate the resulting DE with the conditional prior as a quantitative reference. This scaffold supports Reasoning Elicitation for training-free prediction and mechanistic explanation, and Reasoning Synthesis for generating CoT trajectories for downstream fine-tuning.
Performance with open-weight backbones. CombPert also performs strongly with open-weight backbones. On Norman Seen 0/2, Reasoning Elicitation reduces Qwen3-32B’s MSE from 0.540 under few-shot prompting to 0.321 with the same backbone. With Reasoning Synthesis, CombPert with Qwen3-32B achieves lower MSE (0.245 vs. 0.254) and higher direction accuracy (92.1% vs. 88.2%) than GEARS, with comparable PCC (0.791 vs. 0.800). In this setting, GPT-5.4 generates the training trajectories, and Qwen3-32B performs inference.In the overall Norman evaluation, both Qwen3-32B variants and the Qwen3-8B Synthesis variant outperform GEARS on all three metrics. The Qwen3-32B Synthesis variant also outperforms Scouter and scLAMBDA on all three overall metrics. On Replogle Seen 0/2, the open-weight CombPert variants achieve higher direction accuracy than GEARS, while GEARS achieves lower MSE.
Following the evaluation protocol used in prior work [1], our main results report prediction accuracy on the 20 genes with the largest observed absolute DE for each perturbation. This evaluation emphasizes the strongest transcriptional responses, whose direction and magnitude help characterize the effects of the perturbation.
At inference, CombPert takes a perturbation pair and a target gene as input. It can predict responses across the full measured panel by querying each gene directly, without prior DEG screening. Appendix B.9 and Table 14 of the submitted manuscript already report this full-panel evaluation on Norman. Using GPT-5.4-mini, CombPert achieves an overall PCC of 0.711, compared with 0.593 for GEARS and 0.575 for few-shot prompting with the same backbone. On Seen 0/2, CombPert and GEARS achieve comparable PCC (0.550 vs. 0.549). We will highlight this result in the main text and clarify that the top-20 restriction applies to the primary evaluation subset.
[1] Predicting transcriptional outcomes of novel multigene perturbations with GEARS. Nature Biotechnology, 2024.