- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores how to leverage biological prior knowledge from large language models (LLMs) to refine single-cell RNA velocity inference. The authors first construct an evaluation benchmark, TransBench, across three developmental datasets, confirming that LLMs excel at population-level cell-state transition discrimination but underperform traditional methods in fine-grained quantitative prediction and ranking of gene-level dynamics. Motivated by this capability boundary, the authors propose veloLLM, a post-processing refinement framework that modifies the output of a base velocity model by jointly optimizing three constraints: base velocity field reconstruction, intra-cluster local smoothness, and population-level transition prior edge guidance. Improvements in metrics such as cross-boundary directional accuracy (CBdir) and terminal state identification are demonstrated across four single-cell datasets and five base velocity estimators.
Critical strengths
-
This paper constructs the TransBench evaluation benchmark, quantitatively characterizing the capability boundaries of large language models in population-state topology inference versus gene-expression dynamics prediction, and providing empirical evidence for a division of labor between LLMs and quantitative biophysical models.
-
The proposed veloLLM framework adopts a model-agnostic post-processing refinement paradigm that jointly optimizes velocity reconstruction, manifold smoothness, and prior edge-guidance losses, making it compatible with diverse base estimators while preserving local splicing kinetics and improving the biological plausibility of global developmental trajectories.
Critical weaknesses
-
RNA velocity inference itself is fundamentally constrained by underlying biophysical limitations, including constant kinetic rate assumptions, transcriptional bursting, and local k-NN smoothing and projection artifacts (Bergen et al., 2021; Gorin et al., 2022; Marot-Lassauzaie et al., 2022; Zheng et al., 2023). Rather than addressing velocity estimation from the perspective of transcriptional kinetics or measurement noise modeling, this work forcibly aligns boundary velocity vectors toward known population transition edges; because this training loss shares the exact same reference edges with the core evaluation metric (CBdir), the reported directional gains are largely self-fulfilling and suffer from circular evaluation.
-
The methodological originality of the proposed framework is relatively limited. The overall architecture loosely couples standard retrieval-augmented generation (RAG) with a conventional multilayer perceptron (MLP), where the mean-squared error distillation, intra-cluster cosine smoothness, and cross-boundary directional alignment are straightforward heuristic regularizers rather than principled algorithmic innovations for high-dimensional stochastic single-cell manifolds.
-
There is a notable disconnect between the main empirical results and the central methodological claims of the paper. All performance gains reported in Table 3 of the main text rely on manually curated literature transition graphs rather than priors generated end-to-end by LLMs; as shown in Appendix Table 7, replacing the manual graph with LLM-inferred directed edges omits critical developmental transitions and causes the CBdir score to drop sharply (from 0.7285 to 0.4029 or even 0.0252), undermining the core claim that LLMs can reliably serve as autonomous global prior providers.
-
The TransBench benchmark suffers from severe information asymmetry and construction bias across tasks, making it difficult to objectively establish the claimed capability boundary. In macroscopic state-transition tasks, prompts directly expose standardized cell-cluster names, allowing LLMs to achieve high scores purely through semantic memorization (as evidenced by the sharp performance drop in the hidden-state HTE track); conversely, in microscopic gene-ranking tasks (IGR/DGR), prompts not only strip all quantitative expression values but also mask out almost the entire eligible gene universe from the visible context (e.g., masking 100 candidate genes out of 109 total eligible genes in Hematopoiesis), forcing LLMs to blindly guess noisy single-cell expression differences from nearly empty contexts.
References:
Questions for authors
See weaknesses
3 comments
Response to Reviewer L33q (2/3)
W2: Methodological Originality
Thank you for this comment. We respectfully clarify that the main contribution of this work is not a new network architecture, but a division of labor between LLMs and quantitative velocity estimators, established empirically by TransBench and realized by veloLLM. We intentionally keep the refiner simple so that it can be plugged into any base estimator without modification, as demonstrated across five estimators with different modeling assumptions.
The loss terms are not interchangeable heuristics. Each plays a distinct role (MouseBrain, DeepVelo, Table 11):
| Configuration | VeloCoh | ICCoh | CBdir |
|---|---|---|---|
| DeepVelo base | 0.766 | 0.349 | 0.380 |
| Full refinement | 0.875 | 0.662 | 0.823 |
| Without edge loss | 0.890 | 0.636 | 0.461 |
| Without smoothness | 0.757 | 0.380 | 0.805 |
The edge loss determines cross-boundary direction, while the smoothness term determines within-state coherence. In addition, learned refinement substantially outperforms direct interpolation toward the same graph (see W1), which shows that the design of the objective, rather than the graph alone, drives the improvement.
W3: Disconnect Between the Main Results and LLM-Generated Priors
Thank you for pointing out this disconnect. In the revision, Table 3 reports veloLLM (LLM-generated graph) with DeepSeek-generated graphs as the main result on all four datasets and five base estimators, while veloLLM (human-curated graph) is listed for comparison. Averaged over the five base estimators, CBdir is:
| Dataset | Base | veloLLM (LLM-generated graph) | veloLLM (human-curated graph) |
|---|---|---|---|
| MouseBrain | 0.177 | 0.710 | 0.785 |
| Hematopoiesis | 0.104 | 0.444 | 0.464 |
| Dentate Gyrus | 0.520 | 0.786 | 0.786 |
| Neural Crest | 0.134 | 0.639 | 0.696 |
veloLLM (LLM-generated graph) improves CBdir over the matched base field in all 20 dataset–estimator comparisons and, on average, recovers 88–100% of the gain achieved with the human-curated graph.
The CBdir drop cited by the reviewer comes from the generated graphs in the originally submitted Table 7. In the revision, we manually checked the reference edges for all four datasets against the original studies and prior CBdir evaluations, including those in GFG and TIvelo. DeepSeek and Kimi now recover all reference edges on all four datasets (Table 8). With the scVelo base field, their graphs match the human-curated graph on Hematopoiesis, Dentate Gyrus, and Neural Crest, and reach 0.729 and 0.725 on MouseBrain, compared with 0.771 for the human-curated graph (Table 7).
Response to Reviewer L33q (3/3)
W4: Information Asymmetry and Construction Bias in TransBench
Thank you for raising this concern. We address the two tasks separately.
TE and HTE are designed to measure different conditional capabilities. TE provides cell-state names and evaluates whether a model can apply biological knowledge given annotated identities. This is exactly the setting in which veloLLM uses the LLM, since veloLLM relies on annotated cell states. HTE withholds these names and requires the model to infer identities from gene signatures. The TE–HTE gap therefore does not undermine the capability boundary. It is part of it, showing that LLMs are reliable when reasoning over annotated states but less reliable when inferring states from expression, which is why veloLLM does not ask the LLM for expression-level inference. We will make this conditional interpretation explicit in the revision.
IGR and DGR do not present near-empty contexts. In Hematopoiesis, the 109 genes define the eligible ranking-answer universe, whereas the visible context is drawn from a separate 925-gene expression-context universe. After the 100 candidates are removed, context genes are selected from the remaining 825 genes. Across the Hematopoiesis ranking examples, the median context contains 100 genes on each side, and no list is empty. The expression values of candidate genes are withheld by design, since they would directly reveal the answer.
To test whether the lack of quantitative values explains the low ranking scores, we add the observed expression values of all visible context genes to the prompts, keeping the cell pairs, candidate pools, and reference rankings unchanged (GPT-5.6-Sol, P1):
| Dataset | Task | Examples | Original NDCG@20 | With expression values |
|---|---|---|---|---|
| MouseBrain | IGR | 300 | 0.204 | 0.197 |
| MouseBrain | DGR | 300 | 0.248 | 0.241 |
| Mouse Gastrulation | IGR | 2,000 | 0.219 | 0.198 |
Adding quantitative expression values does not improve ranking performance in any setting. The limited gene-level performance is therefore not an artifact of missing context, which supports the granularity gap reported in TransBench.
We greatly appreciate your insightful comments, which will help improve our paper. If our response has successfully addressed your concerns and clarified any ambiguities, we respectfully hope that you consider raising the score. Should you have any further questions or require additional clarification, we would be delighted to engage in further discussion.
Response to Reviewer L33q (1/3)
We thank the reviewer for the detailed and constructive feedback. We apologize that the original presentation was misleading regarding the role of the transition graph, and we are grateful for the opportunity to clarify a key point: in practice, veloLLM always uses an LLM-generated graph. The human-curated graph serves as the evaluation reference, and veloLLM (human-curated graph) is listed in the performance tables only for comparison. We have now completed the experiments with LLM-generated graphs on all datasets and base estimators. Supplementary analyses use a fixed random seed, so their values differ slightly from the multi-run means in Table 3. Below, we address the reviewer's concerns one by one.
W1: Biophysical Limitations of RNA Velocity and Circular Evaluation
Thank you for this important comment. We respond to its three points in turn.
On kinetic limitations. We agree that veloLLM does not model transcriptional kinetics or measurement noise, and we do not claim to. The limitations listed by the reviewer make local velocity estimates unreliable, and different kinetic models can yield conflicting directions on the same data (Ancheta et al., 2026). veloLLM addresses this problem from a different angle, by adding population-level biological knowledge that kinetic modeling alone cannot provide. Advances in kinetic modeling are inherited through the base estimator, and veloLLM improves CBdir for all five base estimators, including dynamical (scVelo), probabilistic (veloVI), and GRN-informed (RegVelo) models.
On forcible alignment. The alignment is soft rather than forced. The MSE term anchors the refined field to the base estimates, and the edge loss aligns each boundary cell with expression displacements toward its own target-state neighbors rather than with a fixed population direction. A neighborhood-level ODE fit score shows that the refined field remains locally compatible with the spliced-RNA kinetic relation (Figure 4, Appendix A.5).
On circular evaluation. We agree that when the training and evaluation edges coincide, CBdir gains are partly self-fulfilling. This is why veloLLM (human-curated graph) is now listed only for comparison. Two experiments show that the gains of veloLLM are not self-fulfilling.
First, we remove the overlap entirely. We refine the same MouseBrain scVelo field five times, each time withholding one transition from the training graph, and evaluate CBdir only on the withheld transition. A control without edge loss retains velocity alignment and smoothness (Appendix A.7, Table 10):
veloLLM improves every withheld transition over both scVelo and the no-edge-loss control, even though the evaluated edge never appears in training.
Second, if the gains came simply from aligning velocities with the graph edges, directly interpolating the base velocity toward the same edges would achieve a similar CBdir. It does not. The table below reports CBdir, with the interpolation strength swept over five values (Appendix A.8, Table 12):
Imposing the edges directly recovers only a small part of the gain, which shows that the improvement comes from how veloLLM integrates the graph with local expression dynamics.