acceptodds
Withdrawn from ICLR 2027

When Evidence Disagrees: Shapley-Supervised Evidence Selection for Perturbation Response Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting gene expression responses to perturbations is essential for elucidating gene function and guiding experimental design. Recent methods draw on paths retrieved from biological knowledge graphs (KGs) and measurements from related experiments to predict unobserved responses. However, signed KG paths can imply opposing effects, while experimental measurements can report different responses across conditions. The usefulness of each path or measurement can also depend on the perturbation, target gene, cell line, other evidence, and the downstream predictor that uses it. To address this challenge, we introduce PertSelect, a predictor-specific evidence selector that learns which retrieved KG paths and measurements to provide to a given predictor. During training, we estimate predictor-specific Shapley values by averaging how adding each evidence candidate to different subsets changes the predictor's probability of the correct response. Its three variants, PertSelect-Joint, PertSelect-Independent and PertSelect-Kind, use ModernBERT to learn contribution scores and retain evidence with positive scores at inference. We evaluate PertSelect with two predictors on CRISPR interference (CRISPRi) Perturb-seq data from four cell lines: Qwen3-8B, which reads KG paths from OmniPath and INDRA/CoGEx together with related experimental measurements, and a voting predictor that aggregates observed responses of the queried target gene in related perturbation experiments. Across 35,170 held-out queries in four evaluation settings, PertSelect-Joint improves macro-F1 over providing all retrieved evidence by 20.3–31.5 percentage points for Qwen and 8.6–14.4 points for voting. It also achieves higher macro-F1 than embedding-similarity ranking, LLM-based pruning and a learned filter in the style of FILCO in every setting for both predictors. For Qwen, most of its gain comes from deciding how many KG paths and measurements to provide, which removes a strong bias toward predicting down-regulation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
perturbation prediction, knowledge graph

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_McQFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an evidence-filtering approach, based on Shapley values and ModernBERT, that selects knowledge-graph paths and experimental measurements for predicting the direction of a target gene's response (up, down, or no change) to a genetic perturbation. The selected evidence is used by two downstream predictors: a fixed LLM-based predictor and a voting-based predictor. The approach is evaluated on a held-out cell line under both in-distribution and out-of-distribution settings, and it shows improvements over baselines in macro-F1 for both predictors.

Critical strengths

The paper explores an important problem, perturbation response prediction, and how to efficiently leverage prior knowledge and experimental measurements.

Critical weaknesses
  • Limited novelty: The contribution appears limited; the approach seems to be a straightforward application of previously proposed ModernBERT-based evidence filtering, with Shapley values used as the training signal.

The experimental evaluation and setup are underwhelming:

  • Computational cost: Shapley values seem computationally expensive relative to the baselines. Without a report of computational efficiency, it is difficult to evaluate the performance improvements.
  • Clarity and writing: The paper should more clearly explain the prior work it leverages and how it is adapted, e.g., the measurement voting predictor used for evaluation. It should also clarify how the experimental measurements and KG paths are processed and passed to both fixed predictors. The writing needs substantial improvement to highlight the motivation, the specific contributions, and why the proposed approach improves over the baselines. The appendix appears LLM-generated, with several references carrying TODO and verified local PDF annotations.
  • Statistical significance and scope: It is unclear why the results do not report statistical significance and why only one cell line is evaluated.
  • Missing baselines: It is unclear why standard perturbation-prediction baselines are not included, both deep learning and simple linear baselines.
Questions for authors

See weaknesses above

Reviewer_wQyeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PertSelect, a predictor-specific framework for selecting knowledge-graph paths and experimental measurements for gene perturbation response prediction. It estimates training-time Shapley contributions from changes in a fixed predictor's probability of the observed response, andtrains ModernBERT selectors to retain evidence with positive predicted contributions. Three variants differ in candidate encoding and selection granularity. Experiments use CRISPRi data from four cell lines and evaluate Qwen3-8B and measurement voting across four settings. Results show improved macro-F1 over all-evidence prediction and selection baselines, while ablations indicate that Qwen's gains largely arise from controlling evidence quantities and correcting response bias

Critical strengths
  1. Selecting useful evidence from heterogeneous, potentially conflicting biological sources is an important practical problem. PertSelect provides a modular approach that explicitly measures evidence utility for a fixed downstream predictor and requires no query response labels at inference.
  2. The study evaluates two distinct predictors and includes supervision ablations, count-matched randomization, class-offset calibration, and paired uncertainty estimates. These analyses reveal both predictive gains and their limitations, including the dominant role of evidence quantities for Qwen.
Critical weaknesses
  1. For Qwen, randomizing evidence identities at learned counts changes macro-F1 by less than 0.005, and gains are similar with and without disagreement. These findings support predictor-specific input adjustment but provide limited evidence that resolving biological conflicts drives the improvement.
  2. Generalization evidence remains narrow. RPE1 is the only held-out cell line and Qwen3-8B the only LLM. Its improvements over FILCO are not statistically resolved in two OOD settings. Whole cell-perturbation-pair holdout within training cell lines is also untested, limiting evidence for transfer beyond the evaluated contexts.
Questions for authors
  1. PertSelect uses observed response labels to train a ModernBERT selector, while Appendix C.5 compares against fixed class-score offsets. The practical benefit over direct supervised prediction remains unclear. How does PertSelect compare with a ModernBERT response classifier trained on the same queries, evidence, and response labels?
  2. It shows that randomizing evidence identities while preserving PertSelect 's per-query counts changes Qwen macro-F1 by less than 0.005. This control inherits the learned counts and does not establish the necessity of candidate-level contribution estimation. How does PertSelect compare with a directly trained evidence-count controller that does not require candidate-level Shapley supervision?
  3. Qwen3-8B is the only evaluated LLM, and providing all evidence makes it predict down-regulation for 84.9% of queries. When separately trained for another LLM with less severe class bias, does PertSelect retain an advantage over the strongest learned filtering baseline?
  4. Appendix A.2 states that ID test queries share their cell-perturbation context with training queries for other target genes. This evaluates missing targets within partially observed experiments. How much of the selection gain remains when complete cell-perturbation pairs in the training cell lines are held out from all training and measurement retrieval?
Reviewer_dbjmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce PertSelect, a ModernBERT-based selector trained under the supervision of downstream predictors' Shapley values.

Critical strengths

This paper is easy to read.

Critical weaknesses

1.The performance gains do not stem from "selecting the right evidence." In Appendix C.3, keeping the per-kind counts chosen by PertSelect-Joint but randomly replacing the specific candidates leaves macro-F1 nearly unchanged (difference ≤ 0.005, with 98.3% of predictions identical). PertSelect-Kind, which only makes kind-level keep/drop decisions, performs comparably to Joint. Moreover, Joint removes all KG paths for 81.4% of queries.

2.The baselines are overly weak, and key comparisons are missing. All-evidence Qwen underperforms a trivial baseline that predicts "unchanged" for every query (≈ 0.25–0.31 macro-F1) across all four settings. The claimed 20–31 point improvement largely amounts to mitigating a strong bias toward predicting down-regulation, while more rigorous baseline comparisons are omitted.

3.The justification for Shapley supervision is insufficient. Shapley targets incur roughly 9× the computational cost of singleton targets, yet the benefits are inconsistent; notably, in the held-out cell setting, Shapley significantly underperforms singleton supervision.

others: The manuscript appears incomplete, still containing TODO notes and empty sections(C.1).

Questions for authors

1.Performance barely changes when the per-kind counts are fixed and specific candidates are randomly replaced. What does per-candidate Shapley supervision contribute beyond a simple strategy that only learns how many candidates of each kind to retain? Could you include a baseline that solely predicts the per-kind counts?

2.Given that all-evidence Qwen underperforms even a trivial "unchanged" predictor, is it a valid primary baseline? Could you provide comparisons against a majority-class baseline, a classifier trained directly on the same inputs, a fine-tuned Qwen, and established domain methods?

3.Shapley targets require roughly 9× the compute of singleton targets, yet yield inconsistent gains and even degrade performance in the held-out cell setting. Under what conditions is this computational overhead justified, and what accounts for the performance drop in the held-out cell setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.