acceptodds
Under review as a conference paper at ICLR 2027

CellVELA: Contextualizing Cell Foundation Models and LLMs via Retrieved Experimental Evidence

Abstract

Cell foundation models and large language models (LLMs) are increasingly adapted for functional genomics, where predictions often depend on cellular context rather than on genes alone. We study how such context should be transferred to unseen biological settings. Cancer vulnerability provides a stringent testbed: the same gene can be essential in one cancer and dispensable in another, yet standard target-ranking metrics strongly reward genes that are broadly essential. We separate this gene-level prior from context-specific discrimination and show that a fixed ranking can score well while carrying no contextual information. Parameter-efficient fine-tuning of the LLM, including a within-target contrast objective, fits the training contexts but remains near chance on held-out cancer lines across changes in model scale, interface, labels, and sampling. We therefore propose CellVELA, which retrieves matched CRISPR measurements as experimental evidence at inference. This restores held-out discrimination, improves with evidence similarity and quantity, vanishes under random retrieval, and recovers canonical lineage-specific dependencies. The same pattern extends to perturbation-response prediction, where retrieval quality matters more than reader complexity. These results suggest that context-specific functional prediction depends not only on model adaptation, but on preserving the right experimental evidence at inference.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Biological Foundation Models, Representation Learning, Functional Genomics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_acuoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CellVELA studies whether cell foundation models and language models distinguish context-specific effects from general gene-level priors. It shows that a fixed target ranking can score well without identifying which contexts depend on a gene. The paper then compares parameter adaptation with retrieval of experimental measurements. On held-out cancer lines and perturbation-response tasks, the tested fine-tuning recipes transfer poorly, while relevant records improve discrimination. Controls vary retrieval similarity, record count, and reader model. The strongest contribution is the evaluation diagnosis and the importance of relevant measurements, rather than evidence that an LLM is necessary.

Critical strengths
  1. The fixed-ranking control in Table 1 exposes a real evaluation shortcut. The within-gene context metric separates broadly essential genes from context-specific dependencies. This helps determine whether a model learns the distinction required by the biological task, rather than merely exploiting a strong gene prior.

  2. Comparisons with the same retrieved records make the retrieval result informative. Similarity voting and learned readers often perform comparably, random retrieval removes the benefit, and quality and quantity of records affect performance. Stronger ridge references are also reported. Together, these controls support the view that access to relevant measurements drives much of the gain.

Critical weaknesses
  1. The adaptation-versus-retrieval comparison leaves supervision coverage as an alternative explanation. The LLM sees a relatively sparse sample of line–gene labels during training, whereas retrieval supplies relevant measurements directly at inference. Appendix D includes useful sampling controls (PDF p. 15), but coverage and optimization are still not fully separated from where information is stored. The evidence is stronger against the tested adaptation recipes than against parametric learning in general.

  2. Transfer without close screened neighbors remains uncertain. Figure 3a already shows increasing performance with retrieved-line similarity, and the limitations acknowledge the need for related screened contexts. What remains unclear is how much of the held-out gain depends on the test lineage being represented in the bank or on its closest neighbors. This limits the demonstrated range of use without invalidating the held-out results.

  3. The near-perfect biological case scores add limited evidence beyond the aggregate evaluation. Appendix G states that genes were selected after inspecting similarity-vote results, before scoring CellVELA's twenty-record predictions (PDF p. 16). Some cases have too few dependent test lines to satisfy the main class-size criterion. The paper labels them as illustrations and reports results on 1,444 eligible genes. Claims about rare dependencies should therefore rely primarily on broader per-gene results.

Questions for authors
  1. How many distinct eligible line–gene pairs are seen over the full fine-tuning run, rather than in one epoch? Does the current evidence distinguish limited supervision coverage from a limitation of parameter adaptation itself?

  2. Beyond the similarity stratification in Figure 3a, does retrieval still help after excluding each test line's closest training neighbors? Can results be separated by whether the bank contains the test lineage? These analyses would clarify the transfer boundary using the existing data.

  3. For the broader gene set from which the cases were selected, how does context AUC vary with the number of dependent test lines? Please distinguish genes meeting the main evaluation criterion from rarer dependencies, so the highlighted examples can be placed in context.

Reviewer_qNtwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how context-specific functional information can be transferred to unseen biological settings. The authors show that conventional cancer target-ranking metrics are strongly influenced by gene-level priors and introduce within-target Context AUC to evaluate contextual discrimination. They compare LLM parameter adaptation with inference-time retrieval of relevant CRISPR measurements. Experiments show that the investigated fine-tuning approaches generalize poorly to held-out cell lines, while retrieval improves prediction. However, simple similarity-based estimators achieve comparable performance. Additional perturbation-response experiments suggest that retrieval quality often matters more than reader complexity.

Critical strengths

1. Meaningful evaluation perspective. The paper identifies an important limitation of conventional target-ranking metrics: a fixed gene ranking achieves 67.5% Rec.@100 while obtaining a Context AUC of 0.500. The within-target evaluation provides a useful perspective for distinguishing gene-level priors from context-specific functional prediction.

2. Informative empirical analysis. The authors systematically investigate training configurations, representations, and retrieval strategies. Comparisons with random retrieval and simple statistical baselines help identify the sources of performance gains, making the findings valuable beyond the proposed framework.

Critical weaknesses

1. The information-placement interpretation is not sufficiently isolated from other factors. The failure of LLM fine-tuning may reflect differences in supervision coverage, training objectives, or input representations rather than a general limitation of parameter adaptation. Appendix D reports sparse per-epoch supervision for individual targets, whereas the ridge baseline uses continuous effects from all available training screens and achieves a selective Context AUC of 0.669. Although these supervision statistics are not directly comparable, they leave an important confounding factor unresolved. More carefully matched comparisons are needed to substantiate the broader information-placement interpretation.

2. The contribution of the LLM component remains unclear. CellVELA performs comparably to similarity-based voting on cancer dependency prediction (0.572 vs. 0.570 with six records; 0.588 vs. 0.595 with twenty records). Simpler readers also match or outperform it under some PerturbQA configurations. While these findings do not diminish the value of the diagnostic analysis, the paper should better clarify the specific contribution of the LLM framework beyond retrieving relevant experimental evidence.

Questions for authors

Q1. Could the authors report the number of unique positive and negative training contexts per target across the entire training process and provide supervision-matched parametric baselines? This would help distinguish limitations of parameter adaptation from differences in supervision.

Q2. Does the LLM provide any consistent advantage over simpler readers given the same retrieved evidence? If not, could the authors clarify its intended role and the paper's contribution as a diagnostic study?

Q3. To what extent do retrieval gains depend on shared lineage structure between training and test cell lines? A lineage-stratified analysis or comparison of same-lineage and cross-lineage retrieval would help clarify the scope of generalization.

1 comment

Thank you for your response and the additional experiments. These additions have alleviated my main concerns. I am therefore raising my rating to 3.

Reviewer_6YsfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how context-specific functional information should reach a system that combines a cell foundation model with an LLM. It compares three routes within one scoring function: a frozen scGPT embedding projected into two soft tokens, LoRA-adapted LLM weights, and "experimental memory" that measure labels of the query gene in the k training lines which are most similar by expression, written into the prompt (CellVELA).

Critical strengths
  • Context dependence of gene function is one of the oldest questions in functional genomics: the same gene can be essential in one lineage and dispensable in another. Making LLM-based systems confront it explicitly, rather than rewarding them for recovering broadly essential genes, is valuable.

  • The evaluation point is useful and well made. Eq. 1 and Prop. 1 make precise something LLM-for-biology target-ranking benchmarks often ignore: per-line top-K overlap is dominated by broadly essential genes. A context-free list ties the best R@100 and beats every fine-tuned LLM

  • The paper offers clean ablation- each comparison changes a single component of Eq. 3. The random-record control has an exact expectation (Prop. 2) that is confirmed empirically. The paper also includes a width-matched random projection, ridge references, source-matched readers on PerturbQA, and dose–response curves in k.

  • The authors report results that cut against their own method like that ridge beats CellVELA, the contrast loss hurts CellVELA and that cell-FM gene tokens are no better than co-expression or a random projection.

Critical weaknesses
  • It doesn’t seems the proposed method adds no measurable value over simple non-LLM estimators. On DepMap, at k=6, CellVELA matches the kNN vote it reads: 0.531 vs 0.533 (standard) and 0.572 vs 0.570 (selective). At k=20 it falls below the vote (0.588 vs 0.595). On PerturbQA, trees match or beat CellVELA for the strongest keys and the best number in the whole study is a linear reader with a Qwen3-Emb-4B key. CellVELA is never run with that key. The Discussion argues that the LLM is valuable as a "single interface" across assays, but no experiment tests this. As written, the evidence supports "retrieving measured labels helps" far more than "CellVELA is a good method".

  • As the paper point - the central difficulty in context-dependent gene function is that many genes are multifunctional. However, the method retrieves with one global similarity σ(r,r′), cosine on standardised whole-transcriptome expression, for every gene. The neighbours are therefore essentially the same for every query, and in practice they are dominated by lineage. This can recover lineage-restricted dependencies, but it should struggle with dependencies driven by a mutation or a narrow pathway state that cuts across lineages.

  • The paper should also situate itself in the bioinformatics literature on this question, rather than mainly in NLP retrieval work. Relevant work includes predictive dependency modelling and tissue- or context-specific gene networks.

  • The negative claim about parameter adaptation is over-generalised. The per-gene ridge also stores context in parameters and is the best method on every context metric, so what fails is this LoRA recipe, not parametric storage. The recipe is under-supervised: per epoch, the median target appears as a non-dependency in only two training lines, against 246 for the ridge. The fine-tuned LLM underfits genes, falling below the gene prior on Rec@100 (57.1 vs 67.5). It also appears to memorise lines: its 0.74 training AUC and the line-level stripes in Fig. 5 point that way. None of the twelve interventions targets regularisation, training length, LoRA rank or validation-based checkpointing. The ridge also benefits from continuous rather than binary labels. On PerturbQA, fine-tuning does transfer some signal (0.553 vs 0.509 zero-shot). Please scope the claim to the tested recipe and add a binary-label ridge and a neural non-LLM baseline, such as DeepDEP.

  • It is unclear whether perturbation data is an inherent part of the method. As far as I can tell, nothing in CellVELA requires interventional data. DepMap screens are CRISPR knockouts, but the method uses them only as binary labels in a context × gene matrix. The same construction would apply to any measured context-specific label, such as expression or drug response.

  • The theoretical component is thin with no explanation for why parameter adaptation fails to transfer, when retrieval should beat a parametric model, or why the LLM reader converges to the vote. The explanatory framing ("information placement", "long tail") comes by analogy to NLP findings, not by derivation or formal test. An empirical analysis paper does not need a theoretical contribution. However, either Appendix A should be presented as metric properties, or a minimal analysis should be added that makes testable predictions.

Questions for authors

See in weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.