- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper propose a method Pathway Enrichment Analysis Curated via Knowledge (PEACK), a post-hoc framework that improve LLM LLM PEA output based on existing structured prior knowledge, like STRING interaction network. It correct the LLM answer with the Dempster-Shafer theory of evidence and Markov random Field (MRF) prior and doesn't need extra LLM calls.
They firstly confirms that collider bias stems directly from label conditioning, and then address this by treating each sub-query response as the output of a lossy information channel. Queries are scored against a structured MRF prior using DST to calculate the outcome-blind informativeness signal. They evaluate their method on 15 datasets and shows gain.
Critical strengths
-
The writing is structured and figure 1 clearly show method. Taking LLM responses as outputs of a lossy channel is interesting and improve them via DST is novel and useful.
-
It doesn't require extra LLM call. Many methods improve the quality without caring about API attempts cost. This paper is more practical.
-
Experiment is solid. They use 15 datasets to verify the method and validate the impact of this bias on the PEA task with ∼ 46,000 NVIDIA B200 GPU-hours
Critical weaknesses
-
The baseline is weak. Baseline only consider GSEA.
-
Ablation study about the parameters is missing.
-
Lack of ablations on the priors and we don't know how to quantify its contribution in the real application if we don't have ground truth.
-
Test results are presented on the aggregated results of 15 datasets. We don't know the details for each one.
Questions for authors
-
The abstract claims that "precision-recall AUC for all four LLMs from 0.14-0.38 to above 0.9" but Table 1 shows lower values. For example, GPT-OSS Exp 1 .147 ± .129 .310 ± .201 .551 ± .288 GPT-OSS Exp 2 .145 ± .121 .393 ± .236 .419 ± .279 GPT-OSS Exp 3 .190 ± .152 .445 ± .248 .439 ± .251 GPT-OSS Exp 4A .141 ± .113 .388 ± .207 .447 ± .251 GPT-OSS Exp 4B .185 ± .148 .454 ± .255 .575 ± .247
-
Any other related method we need to compare?
-
Could you reports GSEA on the same filtered subsets?
-
Is there a risk that the prior information from KEGG/STRING is too closely with the "ground truth,"? Because the prior might be different in different cases. How to quantify its contribution in the real application if we don't have ground truth.
3 comments
Dear Reviewer,
We believe the current rating is based on concerns that we have addressed in full, including the additional ablation study you requested.
Does the reviewer agree that our clarifications resolved the concerns? Is there any open concern left?
Thanks four authors' detailed response. I also checked other reviewers comments. I keep my scores now and am willing to raise the score if errors identified by Reviewers tSsh and Mq78 are fully solved.
First, the method’s natural baselines are the LLMs query strategies ( the experiments without the retrofit correction) since PEACK is a zero-shot method that uses no Pathways-DB as the introduction states at lines 108–110. Beyond that, GSEA is the standard method of the bioinformatics community, used across biological and healthcare research, and there is no other method in comparably wide use. In the revised version we have also added a prior-only baseline (global response, point 4), which we hope addresses the concern.
Our framework only has one changing parameter per cohort - τ. Within a cohort, ranking by mean ∣η∣ and by mean ∣η∣/τ are the same thing since τ is constant per cohort. We performed an ablation study that compares ranking by ∣η∣ and by ∣K−1∣ and added it to the Per-cohort scale parameter subsection in the appendix (subsection A.5). Over all cells the two rankings agree to Spearman 0.87-0.90 (0.998 in Exp1 for DeepSeek and GPT-OSS). On the retained cells they agree almost exactly. The top-10% overlap by 97-100% in every setting, and the top-2% are identical in Exp 1-3 and Exp4B and overlap by 86-89% in Exp4A. The PR-AUC on the two selections differs by at most 0.01 at the top-10% in every setting, and is identical at the top-2% except in Exp4A (at most 0.04). The selection, and therefore every result of Table 1, is not sensitive to τ.
Please refer to the global response, point 3.
Please refer to our answer to W1 above .
We are not entirely sure what is meant by “filtered subsets.” The paper addresses filtration at the level of LLM calls, each of which contains a list of pathways, whereas GSEA operates on genes. Does the reviewer mean that we should work with subsets of genes (referred to as “subqueries” in the paper)?
This is always possible in principle, however, it is not likely in this case: First, we compare PEACK to a baseline that uses the prior alone, with no LLM output. Please refer to the global response, point 4. Moreover, the TCGA benchmark is considered one of the hallmarks of bioinformatics and (also) the most common PEA benchmark (lines 319–320), with STRING being a common source of prior across biology - and still, PEA on TCGA is considered poorly solved. The very basic prior we used here (local density on graph, quantified as MRF) is highly unlikely to be sufficient alone. Finally, and most importantly, our novelty is the retrofit correction. Even if the prior were closely related to the labels, current LLM-based PEA strategies that use this prior could not exploit it without PEACK. For example, Exp 4A selects the genes shown to the LLM using the same STRING/KEGG prior (with Exp 3 ask the llm to pick them as elicitation). Yet is PR-AUC is no higher than that of random control for any of the four LLMs (Table 1). The gain appears only once the PEACK retrofit correction is applied.
Regarding “the prior might be different in different cases.” - notice STRING is not tied to KEGG or TCGA, STRING contains almost any relation between genes known to science (with sufficient evidence, addressed by the "String confidence score"). While it is true that more “rare phenomena” (rather than cancer) might be less represented given lack of human knowledge about them - STRING is still the widest most common source of information for almost any similar task.