acceptodds
Under review as a conference paper at ICLR 2027

Retrofitting LLM-Based Pathway Enrichment Analysis with Dempster-Shafer Theory of Evidence

Abstract

Pathway enrichment analysis (PEA) attaches functional meaning to sets of observed biological features, such as transcriptomic or proteomic measurements, using curated, labeled knowledge bases. Although PEA is a cornerstone of biological discovery, it still relies on relatively simple statistical tests that cannot exploit the flexibility of modern machine learning tools such as large language models (LLMs). Recent attempts to apply black-box LLMs to PEA have had limited success: LLMs struggle to deliver reliable answers over long, heterogeneous inputs, and current remedies (for example, consensus ranking over many partial queries) incur exploding token costs and break the independence assumption required for valid aggregation. Here we present PEACK (Pathway Enrichment Analysis Curated via Knowledge), a post hoc PEA method that corrects black-box LLM responses using common prior knowledge, such as the STRING interaction network (almost any structured prior suffices), and requires no additional LLM calls. Identifying the core failure of partial-query methods as a "Collider Bias", we correct this bias with the Dempster-Shafer theory of evidence, which represents the uncertainty left by a partial query explicitly rather than counting it as evidence against a pathway. We evaluate four frontier open-source LLMs under five PEA query setups on 15 datasets. PEACK significantly outperforms all existing baselines, raising precision-recall AUC for all four LLMs from 0.14-0.38 to above 0.90, which to our knowledge is the best reported PEA performance for those datasets.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Pathway enrichment analysis, llm

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_YBQPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper propose a method Pathway Enrichment Analysis Curated via Knowledge (PEACK), a post-hoc framework that improve LLM LLM PEA output based on existing structured prior knowledge, like STRING interaction network. It correct the LLM answer with the Dempster-Shafer theory of evidence and Markov random Field (MRF) prior and doesn't need extra LLM calls.

They firstly confirms that collider bias stems directly from label conditioning, and then address this by treating each sub-query response as the output of a lossy information channel. Queries are scored against a structured MRF prior using DST to calculate the outcome-blind informativeness signal. They evaluate their method on 15 datasets and shows gain.

Critical strengths
  1. The writing is structured and figure 1 clearly show method. Taking LLM responses as outputs of a lossy channel is interesting and improve them via DST is novel and useful.

  2. It doesn't require extra LLM call. Many methods improve the quality without caring about API attempts cost. This paper is more practical.

  3. Experiment is solid. They use 15 datasets to verify the method and validate the impact of this bias on the PEA task with ∼ 46,000 NVIDIA B200 GPU-hours

Critical weaknesses
  1. The baseline is weak. Baseline only consider GSEA.

  2. Ablation study about the parameters is missing.

  3. Lack of ablations on the priors and we don't know how to quantify its contribution in the real application if we don't have ground truth.

  4. Test results are presented on the aggregated results of 15 datasets. We don't know the details for each one.

Questions for authors
  1. The abstract claims that "precision-recall AUC for all four LLMs from 0.14-0.38 to above 0.9" but Table 1 shows lower values. For example, GPT-OSS Exp 1 .147 ± .129 .310 ± .201 .551 ± .288 GPT-OSS Exp 2 .145 ± .121 .393 ± .236 .419 ± .279 GPT-OSS Exp 3 .190 ± .152 .445 ± .248 .439 ± .251 GPT-OSS Exp 4A .141 ± .113 .388 ± .207 .447 ± .251 GPT-OSS Exp 4B .185 ± .148 .454 ± .255 .575 ± .247

  2. Any other related method we need to compare?

  3. Could you reports GSEA on the same filtered subsets?

  4. Is there a risk that the prior information from KEGG/STRING is too closely with the "ground truth,"? Because the prior might be different in different cases. How to quantify its contribution in the real application if we don't have ground truth.

3 comments

W1. The baseline is weak. Baseline only consider GSEA.

First, the method’s natural baselines are the LLMs query strategies ( the experiments without the retrofit correction) since PEACK is a zero-shot method that uses no Pathways-DB as the introduction states at lines 108–110. Beyond that, GSEA is the standard method of the bioinformatics community, used across biological and healthcare research, and there is no other method in comparably wide use. In the revised version we have also added a prior-only baseline (global response, point 4), which we hope addresses the concern.

W2. Ablation study about the parameters is missing.

Our framework only has one changing parameter per cohort - τ\tau. Within a cohort, ranking by mean ∣η∣|\eta| and by mean ∣η∣/τ|\eta|/\tau are the same thing since τ\tau is constant per cohort. We performed an ablation study that compares ranking by ∣η∣|\eta| and by ∣K−1∣‾\overline{|K-1|} and added it to the Per-cohort scale parameter subsection in the appendix (subsection A.5). Over all cells the two rankings agree to Spearman 0.870.87-0.900.90 (0.9980.998 in Exp1 for DeepSeek and GPT-OSS). On the retained cells they agree almost exactly. The top-10%10\% overlap by 9797-100%100\% in every setting, and the top-2%2\% are identical in Exp 1-3 and Exp4B and overlap by 8686-89%89\% in Exp4A. The PR-AUC on the two selections differs by at most 0.010.01 at the top-10%10\% in every setting, and is identical at the top-2%2\% except in Exp4A (at most 0.040.04). The selection, and therefore every result of Table 1, is not sensitive to τ\tau.

Q1. The abstract claims that ``precision-recall AUC for all four LLMs from 0.14-0.38 to above 0.9'' but Table1 shows lower values. For example, GPT-OSS Exp 1 .147 ±\pm .129 .310 ±\pm .201 .551 ±\pm .288 GPT-OSS Exp 2 .145 ±\pm .121 .393 ±\pm .236 .419 ±\pm .279 GPT-OSS Exp 3 .190 ±\pm .152 .445 ±\pm .248 .439 ±\pm .251 GPT-OSS Exp 4A .141 ±\pm .113 .388 ±\pm .207 .447 ±\pm .251 GPT-OSS Exp 4B .185 ±\pm .148 .454 ±\pm .255 .575 ±\pm .247

Please refer to the global response, point 3.

Q2. Any other related method we need to compare?

Please refer to our answer to W1 above .

Q3. Could you reports GSEA on the same filtered subsets?

We are not entirely sure what is meant by “filtered subsets.” The paper addresses filtration at the level of LLM calls, each of which contains a list of pathways, whereas GSEA operates on genes. Does the reviewer mean that we should work with subsets of genes (referred to as “subqueries” in the paper)?

Q4. Is there a risk that the prior information from KEGG/STRING is too closely with the "ground truth,"? Because the prior might be different in different cases. How to quantify its contribution in the real application if we don't have ground truth.

This is always possible in principle, however, it is not likely in this case: First, we compare PEACK to a baseline that uses the prior alone, with no LLM output. Please refer to the global response, point 4. Moreover, the TCGA benchmark is considered one of the hallmarks of bioinformatics and (also) the most common PEA benchmark (lines 319–320), with STRING being a common source of prior across biology - and still, PEA on TCGA is considered poorly solved. The very basic prior we used here (local density on graph, quantified as MRF) is highly unlikely to be sufficient alone. Finally, and most importantly, our novelty is the retrofit correction. Even if the prior were closely related to the labels, current LLM-based PEA strategies that use this prior could not exploit it without PEACK. For example, Exp 4A selects the genes shown to the LLM using the same STRING/KEGG prior (with Exp 3 ask the llm to pick them as elicitation). Yet is PR-AUC is no higher than that of random control for any of the four LLMs (Table 1). The gain appears only once the PEACK retrofit correction is applied.

Regarding “the prior might be different in different cases.” - notice STRING is not tied to KEGG or TCGA, STRING contains almost any relation between genes known to science (with sufficient evidence, addressed by the "String confidence score"). While it is true that more “rare phenomena” (rather than cancer) might be less represented given lack of human knowledge about them - STRING is still the widest most common source of information for almost any similar task.

Dear Reviewer,

We believe the current rating is based on concerns that we have addressed in full, including the additional ablation study you requested.

Does the reviewer agree that our clarifications resolved the concerns? Is there any open concern left?

Thanks four authors' detailed response. I also checked other reviewers comments. I keep my scores now and am willing to raise the score if errors identified by Reviewers tSsh and Mq78 are fully solved.

Reviewer_Mq78OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PEACK, a post-hoc filtering framework for LLM-based pathway enrichment analysis. It uses biological structural priors modeled by Markov Random Fields (MRFs) and a Dempster-Shafer Theory (DST)-motivated informativeness score to select pathway-dataset pairs without additional LLM calls. Experiments on 15 TCGA datasets, four LLMs, and five query settings show substantially higher PR-AUC on selected subsets, with additional analyses of query-budget efficiency.

Critical strengths
  1. Interesting approach. PEACK uses external biological knowledge to identify potentially reliable LLM predictions without relying on model-generated confidence or additional inference calls. This offers a useful perspective on selective prediction.

  2. Extensive evaluation. Experiments across multiple LLMs, datasets, and query settings consistently show higher PR-AUC on subsets selected by the proposed score, suggesting that structural priors may help identify more reliable prediction subsets.

Critical weaknesses

1. Inconsistencies in the theoretical derivation.

Eq. (1) is inconsistent with the negative-event odds derivation in Appendix B (p. 19), which implies

Bel(Ei=0∣g)=11+pi(1−pi)K1,t.\mathrm{Bel}(E_i=0\mid g)=\frac{1}{1+\frac{p_i}{(1-p_i)K_{1,t}}}.

Under the maximum-entropy prior pi=0.5p_i=0.5, this correction yields zero uncommitted mass, contradicting Eq. (6). Furthermore, the proposed m(Θ)=(K1,t−1)/(K1,t+1)m(\Theta)=(K_{1,t}-1)/(K_{1,t}+1) becomes negative when K1,t<1K_{1,t} < 1, violating standard DST requirements. Appendix B also appears to conflate Gibbs configuration probabilities with conditional probabilities under partial observations. These issues undermine the theoretical justification of the proposed score.

2. Insufficient validation of the reported performance improvements.

Table 1 reports PR-AUC on selected pathway-dataset subsets rather than the original full evaluation pool. Since PR-AUC depends on class prevalence, the improvements may partly reflect selection effects rather than more reliable LLM discrimination. Positive-label prevalence, matched-coverage random selection, and structural-prior-only baselines are needed.

Additionally, the abstract claims that all four LLMs exceed 0.90 PR-AUC, whereas the best results for Llama-4-Scout and GPT-OSS are only 0.738 and 0.575, respectively.

Questions for authors

Q1. Could the authors provide a mathematically consistent DST derivation, particularly addressing the inconsistency in Eq. (1) and the potentially negative ignorance mass?

Q2. In Appendix B, why can the conditional likelihood under partial observations be computed using Gibbs configuration probability ratios without marginalizing over unobserved features?

Q3. Could the authors report positive-label prevalence and sample counts in the selected subsets, together with matched-coverage random selection and structural-prior-only baselines, to clarify the source of the PR-AUC improvements?

4 comments

We thank the reviewer for the careful reading, and for recognizing the approach as interesting and the evaluation as extensive.

w1. Eq.(1) is inconsistent with the negative-event odds derivation in Appendix B , which implies Bel(Ei=0∣g)=11+pi(1−pi)K1,tBel(E_i=0\mid g)=\frac{1}{1+\frac{p_i}{(1-p_i)K_{1,t}}} Under the maximum-entropy prior pi=0.5p_i=0.5, this correction yields zero uncommitted mass, contradicting Eq.(6). Furthermore, the proposed m(Θ)=(K1,t−1)/(K1,t+1)m(\Theta)=(K_{1,t}-1)/(K_{1,t}+1) becomes negative when K1,t<1K_{1,t}<1, violating standard DST requirements.

Please refer to the global response, points 1 and 2.

Appendix B also appears to conflate Gibbs configuration probabilities with conditional probabilities under partial observations.

We model over gene lists, and a partial query is a complete list: P(gt∣Ei)P(g_t\mid E_i) is the probability of the query that was actually shown, and every gene not in it is withheld from the model. The model has no variable for genes outside the list, so , from the model POV, a partial query is a complete observation, rather than a partial observation of a larger state ( and there is nothing to marginalize over.) eΔΦ(I)−ΔΦ(gt)e^{\Delta\Phi(I)-\Delta\Phi(g_t)} is the ratio of the likelihood ratios of the two exact lists, in which the partition functions cancel, and Proposition B.1 relates the posterior given exactly II to the posterior given exactly gtg_t; the text now defines K1,tK_{1,t} this way. We edited lines 934-936, (in the revised version). We hope that adresses you concerns.

w2. Table1 reports PR-AUC on selected pathway-dataset subsets rather than the original full evaluation pool. Since PR-AUC depends on class prevalence, the improvements may partly reflect selection effects rather than more reliable LLM discrimination.

Respectfully, we believe there may have been a misunderstanding. Table 1 reports evaluation both on the full pool (“Full pool” column), and on the filtered cells by the statistic ∣K−1∣‾\overline{|K-1|}. PR-AUC is computed against all gold-positive pathways in the universe, so it already accounts for both. Precision reflects the false positives among the retrieved pathways. Recall is measured against every true pathway, so true pathways that are not retrieved are penalized, by definition.

Positive-label prevalence, matched-coverage random selection, and structural-prior-only baselines are needed.

Thank you for the comment! we added the true label prevalance in appendix subsection A.6, the average prevalance is 0.143. Therefore, our PR-AUC performence is not affected by the true label prevalance. In addition, We added prior-only baseline as was mentioned in the response for all reviewers (global reponse point 4). Matched-coverage random analysis is added in appendix subsection A.7. PEACK exceeds the matched random baseline in 3838 of the 4040 comparisons. At the top-10%10\% slice it does so in all 2020 settings, and at the top-2%2\% slice in 1818 of the 2020.

Additionally, the abstract claims that all four LLMs exceed 0.90 PR-AUC, whereas the best results for Llama-4-Scout and GPT-OSS are only 0.738 and 0.575, respectively.

Please refer to the global response, point 3.

Q1. Could the authors provide a mathematically consistent DST derivation, particularly addressing the inconsistency in Eq.(1) and the potentially negative ignorance mass?

Please refer to our answer to W1 above .

Q2. In Appendix B, why can the conditional likelihood under partial observations be computed using Gibbs configuration probability ratios without marginalizing over unobserved features?

Please refer to our answer to W1 above .

Q3. Could the authors report positive-label prevalence and sample counts in the selected subsets, together with matched-coverage random selection and structural-prior-only baselines, to clarify the source of the PR-AUC improvements?

Please refer to our answer to W2 above .

Thank you for the detailed response and your efforts to revise the manuscript. I appreciate the corrections to the derivation and the additional experiments. The revised construction addresses the specific algebraic and non-negativity issues raised in my review.

My main remaining concern is the connection between the revised DST formulation and the implemented score. Appendix C.2 motivates the latter through a first-order approximation around K=1K=1 and acknowledges that it may rank pathway–dataset pairs differently from the aggregated DST mass. However, it remains unclear whether the approximation is sufficiently accurate or the stated ranking-preservation conditions hold for the evaluated data, particularly for the selected pairs.

Since this connection is central to the paper's theoretical justification, my concern is not yet sufficiently addressed. I therefore maintain my current rating, while acknowledging the authors' efforts and the improvements made in the revision.

Thank you for the follow-up, and for confirming that the algebraic and non-negativity issues are resolved.

While it is correct that our records are not close to K = 1, and that a first-order statement alone does not cover them, we believe there is some lack of clarity regarding how the score relates to the DST mass, the theory behind that relation, and what should be concluded from it. The relation we rely on is stronger than a first-order expansion, and it is already stated in C.2:

Per record, the two give the same ordering. m_t(Ω) is a strictly increasing function of |K_t − 1| on each side of K = 1, for every K, not only near 1. Ranking needs only this, not numerical closeness.

Hence, the two can order cells differently in only two cases: either (a) the cells being compared lie on opposite sides of K = 1, or (b) K varies a lot between the records of the same cell. However:

Case (a) simply reflects a stated property of the score, that at equal mass it favors K > 1 (C.2), and should be address only as such. This would change the retained cells only if cells strongly on the K < 1 side were among the most displaced, they are not: the τ-free signal |η| of Table 5 treats both directions alike, exactly as the mass does, and retains the same cells as |K−1|.

Case (b) is limited by how K is built. η_t = ΔΦ(I) − ΔΦ(g_t). The first term is the same for every record of a cell; only the second changes with the shown subset.

Table 5 tests both cases on the retained cells. It replaces the per-record transform in the score, which is exponential in η and direction-sensitive, with one that is linear and symmetric in direction. If direction or within-cell spread mattered, this would reshuffle the top cells, and again It does not: the top-10% overlap is 97–100% in every setting, and the top-2% is identical except in Exp 4A (86–89%). PR-AUC changes by at most 0.01 at the top-10% and at most 0.04 at the top-2%. The DST mass is a third transform of the same η_t.

For better clarity, we will rewrite C.2 to lead with the ordering argument instead of the first-order expansion, and to state plainly that the evaluated records are not near K = 1.

With this, we believe the remaining concern is about how C.2 presented the relation, not about the soundness of the method or of any reported result, and we hope the reviewer will weigh it accordingly.

Dear reviewer,

We believe the current rating is based on one remaining concern, which we believe we addressed in full. As the reviewer explicitly stated:

"My main remaining concern is the connection between the revised DST formulation and the implemented score. […] it remains unclear whether the approximation is sufficiently accurate or the stated ranking-preservation conditions hold for the evaluated data, particularly for the selected pairs."

As we explained in our previous comment, the relation we rely on is stronger than a first-order approximation: per record, the DST mass is strictly increasing in ∣Kt−1∣|K_t-1| on each side of K=1K=1, for every KK and not only near 1, so the two give the same per-record ordering. On the evaluated data, Table 5 shows that a direction-symmetric transform retains the same cells (97–100% overlap at the top-10%; identical at the top-2%).

Does the reviewer agree that the ordering argument and Table 5 resolve this concern? Is there any open concern left?

Reviewer_XiY3OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors developed PEACK, a post hoc method for correcting LLM responses in PEA. They identify a specific weakness of the common strategy of splitting a long gene list into partial queries and aggregating the answers. Because each query is implicitly conditioned on the evaluated pathway, confidence is inflated, and so is the false positive rate.

To this end, each partial response is treated as an incomplete observation rather than a vote, using Dempster-Shafer theory so that the uncertainty left by a partial query is represented explicitly rather than counted against a pathway. A Markov random field over STRING with KEGG supplies the prior. The resulting statistic depends only on which genes were withheld, so it needs no additional model calls.

PEACK is evaluated on 15 cancer datasets with four open-weight models across five gene-context regimes, and compared with GSEA.

Critical strengths

The main strength to my understanding is the framing of the problem. Anyone who routinely runs enrichment analysis knows that a pathway is often scored down simply because the relevant genes were not in the submitted list, and that separating that from genuine evidence against the pathway is the right instinct. That the correction requires no additional model calls which is a practical advantage.

Critical weaknesses

My background is in computational biology rather than pure math, so I have focused on the enrichment task and the reported results and left the Dempster-Shafer formulation to other reviewers. One number in the abstract did not match the table as I read it, and I have asked about it below. Also, I wonder if the same results repeat when applying this tool to other omics (if possible).

Questions for authors
  1. The abstract states that PR-AUC rises above 0.90 for all four LLMs. In Table 1, the best top-2% entries I can find for Llama-4-Scout and GPT-OSS are 0.738 and 0.575, while DeepSeek and Qwen3 do exceed 0.90. Have I misread the column?
  2. I wonder whether this approach could be applied to other omics data types, for example, shotgun metagenomics (perhaps using human microbiome datasets?). If so, I think that the value of this work could increase significantly.
1 comment

We thank the reviewer for the review and the favorable score!

Q1. The abstract states that PR-AUC rises above 0.90 for all four LLMs. In Table~1, the best top-2% entries I can find for Llama-4-Scout and GPT-OSS are 0.738 and 0.575, while DeepSeek and Qwen3 do exceed 0.90. Have I misread the column?

Please refer to the global response, point 3.

Q2. I wonder whether this approach could be applied to other omics data types, for example, shotgun metagenomics (perhaps using human microbiome datasets?). If so, I think that the value of this work could increase significantly.

Yes, it is possible. The simplicity of our method is that it needs a labeled universe and a structural prior over features, nothing in it is specific to transcriptomics. So, for shotgun metagenomics the observables are KEGG orthologs, candidate entities are KEGG modules or pathways, and the prior is the ortholog–module membership together with KEGG reaction adjacency, in place of STRING. The taxonomic profile would use taxa as observables and a phylogenetic or co-occurrence network as the prior, with candidate entities defined by curated disease–taxon associations.

Reviewer_tSshOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper address pathway enrichment analysis with black-box LLMs. the authors argue that LLMs struggle with long gene lists, and that querying partial gene subsets and aggregating the answers raises recall but leaves precision low. they attribute this to collider bias, where in multi-label settings, conditioning sub-queries on a candidate label inflates confidence across all labels. They propose PEACK, a post-hoc filter that uses a network prior built from STRING and KEGG to score the LLM's prediction, keeping only the highest-scoring pathways. The method is framed with Dempster-Shafer theory. The authors evaluate it on 15 TCGA cancer cohorts with four self-hosted open-weight LLMs and five gene-selection setups, using MalaCards pathways as ground truth and GSEA as a baseline. They report that restricting to the top 10% or 2% of pathways by this score raises PR-AUC from 0.14–0.38 to as high as 0.95.

Critical strengths
  • The paper proposes ranking LLM predictions by a score computed only from prior network knowledge and the gene subsets shown, without observing the model's outputs or making additional calls. the proposed method is cheap, and can be used on any black-box model.
  • The paper documents a specific failure mode of partial-query LLM PEA. with GPT-5.1, recall rises with more genes while precision stays near 0.2. the main experiments also confirm low unfiltered performance w.r.t. PR_AUC. which makes the paper's motivation credible.
Critical weaknesses
  • The dempster-shafer derivation is wrong. Solving the paper's own odds equation in the proof of Proposition B.1 gives Bel(E=0∣g)=1/(1+p(1−p)K)\mathrm{Bel}(E=0 \mid g) = 1 / \left(1 + \frac{p}{(1-p)K}\right), but Eq. (1) states it with KK multiplying instead of dividing. With the correct form, Bel(E=1∣g)+Bel(E=0∣g)=1\mathrm{Bel}(E=1 \mid g) + \mathrm{Bel}(E=0 \mid g) = 1 for all pp and KK, so the ignorance mass m(Θ)m(\Theta) is always zero. With the stated form, the beliefs sum to more than 1 whenever K<1K < 1. For example, at p=12p = \tfrac{1}{2} and K=13K = \tfrac{1}{3}, both beliefs are 0.750.75 and m(Θ)=−0.5m(\Theta) = -0.5, which is not a valid mass. More fundamentally, the proof treats Bel/(1−Bel)\mathrm{Bel}/(1-\mathrm{Bel}) as ordinary odds and updates them with Bayes' rule, which assumes zero ignorance from the start. In addition, under the paper's own interpretation, a high ∣K−1∣‾\overline{|K-1|} means high ignorance, yet PEACK keeps those cells as the most reliable. This is consequential because DST is presented as the paper's main contribution (title, abstract, Section 3, Appendices B–C). Without it, ∣K−1∣‾\overline{|K-1|} is a heuristic MRF score with no theoretical justification.
  • The evaluation cannot show that PEACK improves LLM-based PEA. All results are PR-AUC computed on the top 10% or 2% of cells chosen by the method itself, compared against GSEA and unfiltered LLM outputs scored on all pathways. The paper doesn't report the positive rate in the filtered subsets, how many true pathways are discarded, or confidence intervals, even though the top-2% subsets contain only a few cells per cohort. Most importantly, it doesn't test ∣K−1∣‾\overline{|K-1|} alone as a predictor. Since ∣K−1∣‾\overline{|K-1|} is essentially a STRING/KEGG network score, the gains may come from the prior rather than the LLM. The abstract's claim of PR-AUC "above 0.90" for all four LLMs is also contradicted by Table 1 (top-2% maxima of about 0.740.74 for Llama-4-Scout and 0.580.58 for GPT-OSS). This is consequential because the paper's empirical claims, including "best reported PEA performance," rest entirely on these comparisons, and without prior-only, same-subset, and coverage analyses, the improvement can't be attributed to the method.
Questions for authors
  • The abstract states that PR-AUC rises above 0.90 for all four LLMs, but Table 1 reports top-2% maxima of about 0.74 (Llama-4-Scout) and 0.58 (GPT-OSS). Could the authors clarify which results support this claim?
  • Exp 2 uses random subsamples that do not condition on any label, yet its low precision is cited as confirming collider bias from label conditioning. How does this result support the collider-bias explanation?
  • how should a negative mass be interpreted?
  • What PR-AUC is obtained by ranking (pathway, cohort) cells by ∣K−1∣‾\overline{|K-1|} (or η\eta) alone, without any LLM output? This would show whether the gains come from the LLM or from the STRING/KEGG prior.
  • Appendix C interprets m(Θ)m(\Theta) as the share of belief committed to neither hypothesis, so a large ∣K−1∣‾\overline{|K-1|} indicates high ignorance. Why, then, are the cells with the highest ∣K−1∣‾\overline{|K-1|} selected as the most reliable?
5 comments

W1. The Dempster-shafer derivation is wrong. Solving the paper's own odds equation in the proof of Proposition B.1 gives Bel(E=0∣g)=1/(1+p(1−p)K)\mathrm{Bel}(E=0\mid g)=1/\big(1+\tfrac{p}{(1-p)K}\big), but Eq. (1) states it with KK multiplying instead of dividing. With the correct form, Bel(E=1∣g)+Bel(E=0∣g)=1\mathrm{Bel}(E=1\mid g)+\mathrm{Bel}(E=0\mid g)=1 for all pp and KK, so the ignorance mass m(Θ)m(\Theta) is always zero. With the stated form, the beliefs sum to more than 1 whenever K<1K<1. For example, at p=12p=\tfrac12 and K=13K=\tfrac13, both beliefs are 0.750.75 and m(Θ)=−0.5m(\Theta)=-0.5, which is not a valid mass. More fundamentally, the proof treats Bel/(1−Bel)\mathrm{Bel}/(1-\mathrm{Bel}) as ordinary odds and updates them with Bayes' rule, which assumes zero ignorance from the start.

Please refer to the global response, points 1 and 2.

In addition, under the paper's own interpretation, a high ∣K−1∣‾\overline{|K-1|} means high ignorance, yet PEACK keeps those cells as the most reliable.

The anchor of the paper is the Dempster–Shafer treatment of a partial query, that explicitly quantifies model ignorance - which in our case translates to "the genes hidden from the model". The structural prior tells us how much those hidden genes matter for pathway ii: the key point is that this ignorance belongs to the individual record, not to the pathway. A pathway whose records repeatedly withhold strong label-discriminating structure is a pathway whose status the cohort's genes strongly determine under the prior - ranking by ∣K−1∣‾\overline{|K-1|} therefore selects pathways on which the question is well-posed.

In detail: for record tt, mt(Θ)m_t(\Theta) is the width of the belief interval [Bel,Pl][\mathrm{Bel},\mathrm{Pl}] about EiE_i and is non-negative. Under the maximum-entropy prior it depends only on the strength of the evidence the structural prior assigns to the withheld genes. PEACK uses this strength but not its direction, which is exactly what leaves the record with an interval of that width.

W2. The evaluation cannot show that PEACK improves LLM-based PEA. All results are PR-AUC computed on the top 10% or 2% of cells chosen by the method itself, compared against GSEA and unfiltered LLM outputs scored on all pathways.

Respectfully, we believe there is a substantial misunderstanding of the paper and its evaluation.

The filtration is part of the method, not a choice of evaluation subset. The theoretical analysis (Sec. 3.1.2) yields an outcome-blind statistic, ∣K−1∣‾\overline{|K-1|}, computed from the structural prior and the shown gene subsets alone (lines 298–301), then, PEACK applies it as a filtration rule. The evaluation itself is standard: pathways are ranked and scored against the MalaCards gold labels over the entire universe of candidate pathways, exactly as for GSEA and the unfiltered LLM outputs.

The paper doesn't report the positive rate in the filtered subsets, how many true pathways are discarded,

PR-AUC is computed against all gold-positive pathways in the universe, so it already accounts for both. Precision reflects the false positives among the retrieved pathways. Recall is measured against every true pathway, so true pathways that are not retrieved are penalized, by definition. About "filtered subsets, how many true pathways are discarded." - again, this is not a metric that requires a report. It is the method itself that rank and filter llm calls, those calls contain many pathways, shared or not with the ones that pass the filtration - the "true pathways are discarded" are reported as any other multiclass problem - by precision recall, in our case the AUC of the P-R graph.

or confidence intervals, even though the top-2% subsets contain only a few cells per cohort.

Table 1 reports mean ± SD over the 15 datasets, and each dataset-level score already aggregates the repeated runs of each arm: 3 repetitions in Exp 1, 500 stratified subsamples in Exp 2, and 2 repetitions per label in Exp 3, 4A and 4B (lines 337–352; App. A.2). Each repetition results with a full answer from the llm (that is, all pathways are considered). A seed-level confidence interval over an already heavily aggregated score (hundred each time) doesn't make much sense, and would require re-running each arm when the existing grid alone required about 46,000 B200 GPU-hours (lines 94–95, 506–507), on the order of hundreds of thousands of dollars in GPU cost, and we believe this is not justified for that purpose.

Most importantly, it doesn't test ∣K−1∣‾\overline{|K-1|} alone as a predictor. Since ∣K−1∣‾\overline{|K-1|} is essentially a STRING/KEGG network score, the gains may come from the prior rather than the LLM.

Please refer to the global response, point 4: we added the prior-only baseline to Table 1.

The abstract's claim of PR-AUC "above 0.90" for all four LLMs is also contradicted by Table 1 (top-2% maxima of about 0.74 for Llama-4-Scout and 0.58 for GPT-OSS).

Please refer to the global response, point 3.

Q1. The abstract states that PR-AUC rises above 0.90 for all four LLMs, but Table 1 reports top-2% maxima of about 0.74 (Llama-4-Scout) and 0.58 (GPT-OSS). Could the authors clarify which results support this claim?

Please refer to the global response, point 3.

Q2. Exp 2 uses random subsamples that do not condition on any label, yet its low precision is cited as confirming collider bias from label conditioning. How does this result support the collider-bias explanation?

We agree with the reviewer - the intended hypothesis is that conditioning happens on the shown genes: a gene shared by several pathways is a collider for them, regardless of how it was selected. We refined the text (lines 92-99) accordingly.

Q3. how should a negative mass be interpreted?

Please refer to the global response, point 2.

Q4. What PR-AUC is obtained by ranking (pathway, cohort) cells by ∣K−1∣‾\overline{|K-1|} (or η\eta) alone, without any LLM output? This would show whether the gains come from the LLM or from the STRING/KEGG prior.

Please refer to the global response, point 4.

Q5. Appendix C interprets m(Θ)m(\Theta) as the share of belief committed to neither hypothesis, so a large ∣K−1∣‾\overline{|K-1|} indicates high ignorance. Why, then, are the cells with the highest ∣K−1∣‾\overline{|K-1|} selected as the most reliable?

Please refer to our answer to W1 above .

Thanks for the clarifications. I appreciate the efforts. The revision fixed the algebra and added analyses I requseted. I am willing to raise my score.

Respectfully, we believe there is a substantial misunderstanding of the paper and its evaluation. The filtration is part of the method, not a choice of evaluation subset. The theoretical analysis (Sec. 3.1.2) yields an outcome-blind statistic, ∣K−1∣‾\overline{|K-1|} , computed from the structural prior and the shown gene subsets alone (lines 298–301), then, PEACK applies it as a filtration rule. The evaluation itself is standard: pathways are ranked and scored against the MalaCards gold labels over the entire universe of candidate pathways, exactly as for GSEA and the unfiltered LLM outputs.

I don’t object to evaluating PEACK on the cells it retains, my concern is that PEACK’s scores are computed on the retained subset while the paper’s headline claims compare them with GSEA and unfiltered LLM scores computed on all pathways, and attribute the difference as improvements to PEACK. I believe they should be compared at the same coverage. The new matched random baseline (table 7) in the revised version is the right comparison. It shows that much of the reported gain appears for any subset of the same size. I would ask that the abstract and main claims be stated relative to this baseline.

Comments on the paper

1 comment

We thank all the reviewers for the careful reviews, and are glad to hear the reviewers find the work practical, the problem framing and motivation well-grounded, the approach interesting and novel, and the evaluation extensive and solid. Here we address concerns that were shared by multiple reviewers:

Solving the paper's own odds equation in the proof of Proposition B.1 gives Bel(E=0∣g)=1/(1+p(1−p)K)Bel(E=0\mid g)=1/\left(1+\frac{p}{(1-p)K}\right), but Eq.(1) states it with KK multiplying instead of dividing. With the correct form, Bel(E=1∣g)+Bel(E=0∣g)=1Bel(E=1\mid g)+Bel(E=0\mid g)=1 for all pp and KK, so the ignorance mass m(Θ)m(\Theta) is always zero. With the stated form, the beliefs sum to more than 1 whenever K<1K<1. For example, at p=12p=\frac{1}{2} and K=13K=\frac{1}{3}, both beliefs are 0.750.75 and m(Θ)=−0.5m(\Theta)=-0.5, which is not a valid mass.(Reviewer tSsh)

Eq1 is inconsistent with the negative-event odds derivation in Appendix B , which implies Bel(Ei=0∣g)=11+pi(1−pi)K1,tBel(E_i=0\mid g)=\frac{1}{1+\frac{p_i}{(1-p_i)K_{1,t}}} Under the maximum-entropy prior pi=0.5p_i=0.5, this correction yields zero uncommitted mass, contradicting Eq.(6). Furthermore, the proposed m(Θ)=(K1,t−1)/(K1,t+1)m(\Theta)=(K_{1,t}-1)/(K_{1,t}+1) becomes negative when K1,t<1K_{1,t}<1, violating standard DST requirements. (Reviewer Mq78)

Regarding the theoretical derivation regarding Bel(Ei=0∣g)Bel(E_i=0 | g), indeed, the second formula of Proposition B.1 / Eq. (1) divided the odds for Ei=0E_i=0 by KK instead of 1/K1/K, which was a typo. We are truly sorry for this mistake and we fixed both equations in the revised version.

Regarding the potentially negative ignorance mass: in addition to the typo, there was a missing definition in the paper that defines the uncommitted mass and resulting in the mass being non-negative. We use the strength of the withheld evidence,κ=max⁡(K,1/K)\kappa=\max(K,1/K), and not its direction, because the direction is the structural prior’s verdict on the pathway, which is what the LLM is asked to form. Then, the posterior given gg is known only to lie between the values obtained at κ\kappa and 1κ\frac{1}{\kappa} , and the Dempster–Shafer belief state of the record is their lower envelope. The two beliefs now sum to at most one, and the uncommitted mass is non-negative. The informativeness signal (∣K−1∣‾\overline{|K-1|}) and all reported results are therefore unchanged. We corrected the definition and the mass formula in Section 3.1.2 and Appendix C.1–C.2, thank you again!.

Reviewer tSsh: The abstract's claim of PR-AUC ``above 0.90'' for all four LLMs is also contradicted by Table1 (top-2% maxima of about 0.740.74 for Llama-4-Scout and 0.580.58 for GPT-OSS).

Reviewer XiY3: The abstract states that PR-AUC rises above 0.90 for all four LLMs. In Table1, the best top-2% entries I can find for Llama-4-Scout and GPT-OSS are 0.738 and 0.575, while DeepSeek and Qwen3 do exceed 0.90. Have I misread the column?

Reviewer Mq78: the abstract claims that all four LLMs exceed 0.90 PR-AUC, whereas the best results for Llama-4-Scout and GPT-OSS are only 0.738 and 0.575, respectively.

Reviewer YBQP: The abstract claims that ``precision-recall AUC for all four LLMs from 0.14-0.38 to above 0.9'' but Table1 shows lower values. For example, GPT-OSS Exp 1 .147 ±\pm .129 .310 ±\pm .201 .551 ±\pm .288 GPT-OSS Exp 2 .145 ±\pm .121 .393 ±\pm .236 .419 ±\pm .279 GPT-OSS Exp 3 .190 ±\pm .152 .445 ±\pm .248 .439 ±\pm .251 GPT-OSS Exp 4A .141 ±\pm .113 .388 ±\pm .207 .447 ±\pm .251 GPT-OSS Exp 4B .185 ±\pm .148 .454 ±\pm .255 .575 ±\pm .247

The abstract line is "PEACK significantly outperforms all existing baselines, raising precision-recall AUC for all four LLMs from 0.14-0.38 to above 0.90, ...". The word “all” in the sentence is referring to “raising precision-recall AUC”, which is correct for all four of them. We understand that it might be read differently, so we changed the sentence to “raising precision-recall AUC for the LLMs from 0.14-0.38 to above 0.9.” in the revised version. We hope that adresses the concerns.

(Reviewer tSsh) it doesn't test ∣K−1∣‾\overline{|K-1|} alone as a predictor. Since ∣K−1∣‾\overline{|K-1|} is essentially a STRING/KEGG network score, the gains may come from the prior rather than the LLM.

(Reviewer Mq78) the improvements may partly reflect selection effects rather than more reliable LLM discrimination. ..... and structural-prior-only baselines are needed.

(Reviewer YBQP) Lack of ablations on the priors

Thank you for the comment, we added prior-only results to Table 1. As the new results shows, the prior-only results are poor and not comparable or better than the PEACK results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.