acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Study-Level Transcriptomic Hypothesis Generation with Biological Reasoning

Abstract

Grounded biological reasoning on transcriptomic studies remains a major challenge for genomic and large language models. While existing transcriptomic foundation models learn cell- and gene-level representations, they operate primarily on individual cells and struggle to generate mechanistic explanations for study-level phenomena. General-purpose large language models (LLMs) can generate plausible biological narratives, yet they frequently miss the regulatory pathways and molecular cascades driving transcriptomic signatures. We formulate a task where an LLM is given a disease context alongside descriptions of two biological conditions and must predict the intermediate mechanistic cascades separating them. Using curated condition contrasts from RummaGEO, we construct a benchmark and training dataset of 15,251 human and mouse mechanistic reasoning samples. Each sample maps biological conditions to grounded multi-scale cascades of kinases, transcription factors, and biological processes derived from empirical differential expression enrichment. We then train a TRACE (Transcriptomic Reasoning & Analysis of Condition Expression), an open-source 4B-parameter reasoning model based on Qwen3.5, and conduct rubric-based evaluation of hypothesis generation assessing reasoning trace coherence, final mechanistic summaries, and recovery of top enriched regulatory entities. Across held-out GEO studies and unseen disease contexts, TRACE recovers more empirically supported regulatory terms and mechanistic pathways than open-source and frontier model baselines. Case studies confirm that our model generates supported mechanistic cascades in its reasoning. We release the 15k-sample reasoning dataset, the TRACE model, and the enrichment-to-trace synthesis pipeline to facilitate scalable mechanistic reasoning in transcriptomics.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Deep Learning, Transcriptomics, Reasoning, RNA Sequencing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2nLiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a 4B-parameter LLM for study-level mechanistic hypothesis generation from biological condition descriptions. The also present a dataset of 15,251 transcriptomic contrasts, using enrichment-derived kinase–transcription factor–pathway cascades to generate supervised reasoning traces. Evaluation on held-out GEO studies and unseen diseases shows substantial improvements over open-source and commercial LLM baselines in recovering enriched regulatory entities.

The paper is well-written, clearly structured, and addresses an interesting problem with promising empirical results. However, it remains unclear whether the model learns genuine mechanistic reasoning or primarily captures associations between metadata and enrichment patterns.

Critical strengths
  1. The authors benchmark TRACE against several open-source and commercial LLMs across held-out studies and unseen disease contexts, providing meaningful evidence of improvements on the proposed task. The evaluation also includes detailed biological case studies and analyses of individual regulatory entities (e.g., kinases, TFs, and pathways), going beyond aggregate performance metrics.

  2. The authors construct a relatively large dataset of transcriptomic reasoning samples and introduce a systematic pipeline connecting differential-expression enrichment to mechanistic hypothesis generation. The integration of established biological resources and the proposed release of the dataset and pipeline constitute potentially valuable contributions for future research in transcriptomic interpretation and biological reasoning.

Critical weaknesses
  1. The evaluation relies on a single LLM judge (Gemini 3.5 Flash-Lite), which is also used to generate the training reasoning traces. This raises concerns about potential evaluation bias and the robustness of the reported scores. It is unclear whether the relative model rankings would remain consistent under different, potentially stronger or independent evaluators. Comparing multiple LLM judges and reporting their agreement would substantially strengthen confidence in the results.

  2. Although the paper provides detailed biological case studies, only two individual examples are examined in depth, and their selection criteria are not clearly specified. This raises concerns about how representative these examples are of the model's overall performance. More systematic validation, including randomly selected successful and unsuccessful predictions and independent expert assessment of the proposed regulatory cascades, would help establish whether the model generates biologically meaningful mechanisms rather than simply recovering enriched entities.

Questions for authors
  1. What motivated the selection of Gemini 3.5 Flash-Lite as both the teacher model for reasoning-trace generation and the evaluator? Given the potential for teacher–evaluator bias, could the authors validate their results using independent LLM judges or expert biological assessments, and demonstrate that the relative model rankings remain consistent?

  2. The paper presents two qualitative case studies (Brugada syndrome and hyperglycemia), but does not clarify how these examples were selected. Could the authors specify the selection criteria and provide additional randomly sampled examples, including unsuccessful predictions, to assess whether the illustrated mechanistic reasoning is representative of the model's overall performance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2nLi

Weakness 1 / Question 1: Teacher–evaluator bias and independent judges

The evaluation relies on a single LLM judge (Gemini 3.5 Flash-Lite), which is also used to generate the training reasoning traces. This raises concerns about potential evaluation bias and the robustness of the reported scores. It is unclear whether the relative model rankings would remain consistent under different, potentially stronger or independent evaluators. Comparing multiple LLM judges and reporting their agreement would substantially strengthen confidence in the results.

What motivated the selection of Gemini 3.5 Flash-Lite as both the teacher model for reasoning-trace generation and the evaluator? Given the potential for teacher–evaluator bias, could the authors validate their results using independent LLM judges or expert biological assessments, and demonstrate that the relative model rankings remain consistent?

We thank the reviewer for this feedback. We originally selected Gemini 3.5 Flash-Lite for both teacher generation and evaluation because it offered biological reasoning capabilities at a cost that allowed generating and grading tens of thousands of samples. We agree that this creates a risk of teacher–evaluator bias, and we address this concern in detail in the general comment addressed to all reviewers above, with multiple graders and a revised rubric. Under rubric v2, the three graders' model rankings are consistent (Spearman correlations of 0.73–0.93), and we base our main conclusions on the two graders outside the teacher's model family.

We also added two metrics independent of an LLM judge (Tables R3 and R4 in our response to Reviewer Abu1), providing complementary evidence of per-sample reasoning performance from our models.

Weakness 2 / Question 2: Case-study selection and representativeness

Although the paper provides detailed biological case studies, only two individual examples are examined in depth, and their selection criteria are not clearly specified. This raises concerns about how representative these examples are of the model's overall performance. More systematic validation, including randomly selected successful and unsuccessful predictions and independent expert assessment of the proposed regulatory cascades, would help establish whether the model generates biologically meaningful mechanisms rather than simply recovering enriched entities.

The paper presents two qualitative case studies (Brugada syndrome and hyperglycemia), but does not clarify how these examples were selected. Could the authors specify the selection criteria and provide additional randomly sampled examples, including unsuccessful predictions, to assess whether the illustrated mechanistic reasoning is representative of the model's overall performance?

We thank the reviewer for this comment. The original two case studies were selected to illustrate reasoning on held-out diseases. We have added 20 randomly selected case studies in the PDF linked below, including both successful and unsuccessful predictions, so that the representativeness of the illustrated reasoning can be assessed.

Twenty reasoning traces: https://anonymous.4open.science/r/Anonymous_TRACE-DFDE/TRACE_20_supplementary_case_studies.pdf

We are working on obtaining comments from an expert biologist on these reasoning case studies, which we will include in the final submission and, time permitting, during this rebuttal period.

Hi reviewer 2nLi,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

I appreciate the author's response to my concerns and the performed experiments. I've increased my score.

Reviewer_2YHtOpenReview ↗
Rating
4(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a dataset and a 4B model for writing mechanistic hypotheses about GEO studies. The authors pair each of 263 Jensen DISEASES terms with two GO Biological Process terms (89 pairings were corrected by hand) and retrieve RummaGEO signatures that overlap these gene sets. Each signature is enriched against KEA 2015, ChEA 2022 and GO BP 2025, and the top 10 terms per library plus five co-enriched kinase, TF and GO triplets form a hidden label. Gemini 3.5 Flash-Lite sees this label and writes a reasoning trace and a final answer for a prompt that holds only the disease, the two GO terms and the two condition descriptions. Qwen3.5 4B is then fine-tuned on the 15,251 resulting samples (7,100 human and 8,151 mouse, from 3,441 GEO series and 242 diseases).

The same Gemini model grades all outputs against the hidden labels on a 10-point rubric (reasoning 4, triplet alignment 3, list coverage 3). On 2,451 samples from held-out GEO series, TRACE scores 5.90, against 4.28 for Gemini 3.5 Flash-Lite and 3.33 for GPT 6 Luna. On 315 samples from five held-out diseases it scores 4.98 against 3.49. The paper also extracts the entities named in TRACE's answers and compares them with each sample's enrichment (Fig. 4, Fig. S1), and walks through two held-out disease cases, Brugada syndrome and hyperglycemia.

Critical strengths
  1. The task is set up so that answers can be checked. Each hypothesis is graded against what the study's own differential expression showed, a firmer target than plausibility, and the same hidden labels supervise the teacher. The recipe extends to any GEO contrast. The text names Gemini 3.5 Flash-Lite as the teacher, and TRACE beats that model when it runs without the labels (5.90 vs 4.28 on held-out series, 4.98 vs 3.49 on held-out diseases), so the student has picked up something from the labels that the teacher cannot produce from the prompt alone.

  2. The dataset covers 15,251 contrasts from 3,441 GEO series in two species, and the disease and GO pairings were reviewed by hand. Every threshold in the pipeline (Fisher tests, FDR and odds ratio cutoffs, caps per disease and species) is stated, so the pipeline can be rebuilt or extended. Once the data, the model and the pipeline are released as planned, others can build on them directly.

  3. The paper checks TRACE's answers beyond the judge score. Fig. 4 extracts the kinases, TFs and processes named in each answer and compares them with the sample's own enrichment, and in Fig. S1 the named entities shift with the disease system in step with the data (r = 0.73). The case studies are checked against the literature, and the authors point out the AMPK step in the hyperglycemia case that contradicts it.

Critical weaknesses
  1. With no baselines that use only priors, I cannot tell how much of the gain is study specific. The labels concentrate on a few hubs. SRC, PRKCA, GSK3B, PRKACA and MAPK1 are each enriched in roughly half or more of all samples (Fig. 2C), and SUZ12, WT1 and MTF2 in about 40% (Fig. 2D). The authors attribute the held-out series result to the model having seen similar enrichment profiles for the same disease during training (Sec. 4.2). In Fig. S1E, PRKACA appears in 94% of Brugada answers and PRKAA2 in 81% of hyperglycemia answers, and without the enrichment rates for those same samples it is unclear whether this follows the samples or the disease name.

Fig. 4 does not settle the question. The correlation of 0.78 between how often a kinase is named and how often it is enriched is also what a model that learned marginal frequencies would show. The share of named entities found in the sample's own enrichment (59 to 78% in Fig. 4C) has no base rate and is reported for TRACE only. Learning a good disease-level prior would still be worth something, but the paper's claim is about individual studies, and that needs this comparison.

  1. Scoring relies on one LLM judge from the teacher's family, with no check against human ratings. Gemini 3.5 Flash-Lite grades every output, the text names it as the teacher (Fig. 1 says Gemini 3.5 Flash), and it is also the strongest baseline. TRACE's lead is in triplet alignment and list coverage, which are closer to entity matching than the reasoning score, so I expect the lead to hold. The rubric still gives partial credit for gene-family members and related pathway names, and the paper reports no agreement with expert ratings or with a second judge. Running the entity checks of Fig. 4 for all models would answer much of this without a judge.

The paper also leaves out the coherence score defined in A.1.1. About 20% of TRACE outputs have no completed final answer (1,953 of 2,451 and 250 of 315), and it is not said how these are scored or how often the baselines fail. Base Qwen3.5 4B scores 0.10 out of 10, which looks like a format or length failure, and no decoding settings are given. Last, the labels are enrichment results and so a proxy for regulator activity. Describing the predicted cascades as genuine experimental mechanisms (Sec. 1) goes further than this evaluation supports.

  1. The claim that TRACE learns generalizable rules (Sec. 4.2) rests on what is in effect a four-disease split. The four diseases in Fig. S1E account for 248 of the 250 completed held-out disease answers, and the fifth disease is not named. The paper does not say whether GEO series or signatures in this split also appear in training under other diseases. A signature can be retrieved for several diseases that share genes, and a cardiomyocyte differentiation series like the one in the Brugada case could also match other cardiac diseases. Standard errors are computed over samples, which cluster by series and by disease, so with four diseases they understate the uncertainty.
Questions for authors
  1. Please add baselines that use only training priors: for each disease, an answer built from the most frequent kinases, TFs and GO terms in its training samples, and Gemini given the nearest training samples with their labels in context. Please score them with the rubric and with the entity-level precision and recall of Fig. 4, applied to all models. Within a disease, does naming an entity track that sample's enrichment, for example measured by AUROC across the disease's samples?

  2. Could you report agreement between the judge and expert ratings on a few hundred samples, or scores from a second judge from another model family? Please also report the coherence scores, the completion rate of every model, how incomplete outputs are scored, and the decoding settings, including the generation budget for Qwen3.5 4B. Which Gemini model wrote the training traces?

  3. Which is the fifth held-out disease? What fraction of held-out disease samples share a GEO series or a signature with the training set, and how do the results change when those samples are removed? Confidence intervals bootstrapped over diseases and series would also help.

On related work, Sec. 2.2 cites BioReason, and its follow-up BioReason-Pro (bioRxiv, 2026) should be cited as well.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2YHt (Part 1/5)

Weakness 1 / Question 1: Prior-only baselines and sample specificity

Is the gain study-specific? Please add baselines that use only training priors: for each disease, an answer built from the most frequent kinases, TFs and GO terms in its training samples, and Gemini given the nearest training samples with their labels in context. Please score them with the rubric and with the entity-level precision and recall of Fig. 4, applied to all models. Within a disease, does naming an entity track that sample's enrichment, for example measured by AUROC across the disease's samples?

We have added the requested baselines:

  • Global prior: the most frequent kinases, TFs, and GO terms across all training samples.
  • Disease prior: the most frequent entities among training samples of the same disease. This applies to held-out GEO series only, since held-out diseases have no training samples.
  • kNN prior: the top entities among the three most similar training studies, using TF-IDF over study descriptions.
  • Gemini-kNN: Gemini 3.5 Flash-Lite given those three training studies and their enriched kinases, TFs, and GO terms in context.

We set up each prior to predict the same number of entities as TRACE for the corresponding sample. All models are evaluated against the sample's own enrichment, using precision and recall metrics as suggested by the reviewer. We note that on the previous benchmark, where samples were pooled by disease to identify consensus enrichment, a disease prior is effectively an upper ceiling of performance. We report the results below and add them to the revised manuscript in Appendix Table S7.

Table R5: Entity precision (P) and recall (R), in percent, of the kinases, TFs, and GO processes named in each final answer against the sample's own top-10 enrichment. Precision is averaged over samples in which the method names at least one entity of that type; recall is averaged over all samples with a non-empty reference. Priors list as many entities as TRACE (per-sample) named for each sample.

Held-out GEO studies (n = 2,451)

ModelKinase PKinase RTF PTF RGO PGO R
Teacher reference90.542.388.424.882.120.3
Qwen3.5-4B base13.50.812.31.111.61.4
Gemma-3-4B-it17.70.611.70.718.91.8
Llama-3.2-3B-Instruct14.50.511.60.518.91.2
OpenBioLLM-Llama3-8B13.20.08.30.015.40.2
DeepSeek-R1-Distill-Qwen-7B16.00.315.50.616.30.9
Gemini 3.5 Flash-Lite21.20.716.91.416.11.3
GPT-6 Luna12.81.714.52.316.60.9
Global prior43.815.540.36.921.53.5
Disease prior72.426.882.120.677.717.5
kNN prior61.721.972.518.465.715.0
Gemini-kNN (retrieval)59.822.463.626.257.813.8
TRACE 4B (disease-pooled, ours)71.817.475.316.960.112.4
TRACE 4B (per-sample, ours)62.923.068.016.056.212.5

Held-out diseases (n = 315)

ModelKinase PKinase RTF PTF RGO PGO R
Teacher reference90.851.490.133.476.021.2
Qwen3.5-4B base3.80.17.30.83.30.4
Gemma-3-4B-it0.00.06.20.63.70.4
Llama-3.2-3B-Instruct6.20.05.20.32.40.2
OpenBioLLM-Llama3-8B0.00.00.00.08.80.1
DeepSeek-R1-Distill-Qwen-7B5.00.01.80.14.90.3
Gemini 3.5 Flash-Lite0.00.014.61.56.50.3
GPT-6 Luna2.10.27.91.33.60.1
Global prior35.713.545.110.63.70.9
Disease prior––––––
kNN prior55.719.866.217.950.512.3
Gemini-kNN (retrieval)55.924.161.625.838.07.7
TRACE 4B (disease-pooled, ours)60.115.565.518.937.47.4
TRACE 4B (per-sample, ours)51.417.755.612.835.39.0

TRACE's overall precision is substantially higher than that of the general LLMs, demonstrating its strong performance at naming relevant enrichment terms. On holdout GEO series, TRACE (per-sample) also has higher paired kinase and TF precision than Gemini-kNN. TRACE's enrichment precision and recall exceed those of all general LLM baselines across both splits. Disease and retrieval priors remain strong baselines, particularly for TF recall.

(continued in next comment)

AuthorsOpenReview ↗

Response to Reviewer 2YHt (Part 2/5)

Does it follow the sample or the disease name?

To test sample specificity, we use Gene Signature Sample Rank (GSSR) (Table R4, in our response to Reviewer Abu1 above), which measures how well each answer's associated gene set covers its own sample's differentially expressed genes relative to other studies of the same disease. On holdout GEO study samples, TRACE (per-sample) improves over the disease prior by +0.022 and over Gemini-kNN by +0.012. These comparisons support a modest sample-specific signal beyond disease-level signals.

Additionally, when predicted entities are evaluated against external information sources (Table R3), TRACE (per-sample) also improves over the disease prior on DisGeNET in terms of EES, while the Open Targets comparison is comparable.

Per-sample reference

We also constructed a separate reasoning dataset and benchmark where enrichment is computed per sample, not pooled across disease samples. We rerun the analysis with sample-specific enrichment to evaluate whether TRACE performs well on sample-specific signals; in contrast, the original TRACE model (disease-pooled) was trained on enrichment ground truth calculated across samples enriched for a disease, which was intended to identify consistently enriched entities in that disease context. The results for sample-specific evaluation are shown below, with the newly trained TRACE 4B model on sample-specific enrichment denoted as TRACE (per-sample).

Table R6: Rubric v2 evaluation using sample-specific enrichment references. Each cell reports Claude Haiku 5.5 / GPT-6 Luna scores (mean over 670–685 held-out GEO study and 310–315 held-out disease samples per model and grader).

ModelHeld-out GEO series (n = 685)Held-out diseases (n = 315)
GPT-6 Luna5.41 / 5.295.00 / 4.80
Gemini 3.5 Flash-Lite5.18 / 4.965.05 / 4.63
Gemini-kNN (retrieval)6.56 / 6.486.06 / 5.72
TRACE 4B (per-sample, ours)6.07 / 5.965.37 / 5.08

Under both graders and across both splits, TRACE retains higher mean scores than the general LLMs, performing worse only than the retrieval-augmented ceiling (Gemini-kNN). These results show that TRACE's advantage over the general LLMs persists when grading against sample-specific enrichment, rather than pooled disease-level references. Together with the independent-grader and judge-free evaluations, this provides complementary evidence that the reported gains do not depend solely on agreement with pooled disease-level enrichment.

(continued in next comment)

AuthorsOpenReview ↗

Response to Reviewer 2YHt (Part 3/5)

Weakness 2 / Question 2: Judge agreement, coherence, completion, and teacher identity

Scoring relies on one LLM judge from the teacher's family, with no check against human ratings. Gemini 3.5 Flash-Lite grades every output, the text names it as the teacher (Fig. 1 says Gemini 3.5 Flash), and it is also the strongest baseline. TRACE's lead is in triplet alignment and list coverage, which are closer to entity matching than the reasoning score, so I expect the lead to hold. The rubric still gives partial credit for gene-family members and related pathway names, and the paper reports no agreement with expert ratings or with a second judge. Running the entity checks of Fig. 4 for all models would answer much of this without a judge.

Independent graders. We have added GPT-6 Luna and Claude Haiku 5.5 alongside Gemini as grading LLMs, detailed fully in the general comment to all reviewers above. We report agreement metrics between graders in the general comment as well as in the revised manuscript, and we base our main conclusions on the non-teacher graders. We find that TRACE's advantage over general LLMs is supported under both non-teacher graders, which is expected since our benchmark evaluates a model's biological reasoning by multiple criteria, including enrichment entity recall and coherence.

We have added two judge-free metrics (Tables R3 and R4 in our response to Reviewer Abu1), as well as entity precision-recall checks for all models (Table R5).

Weakness 2b: Coherence score and decoding settings

The paper also leaves out the coherence score defined in A.1.1. About 20% of TRACE outputs have no completed final answer (1,953 of 2,451 and 250 of 315), and it is not said how these are scored or how often the baselines fail. Base Qwen3.5 4B scores 0.10 out of 10, which looks like a format or length failure, and no decoding settings are given. Last, the labels are enrichment results and so a proxy for regulator activity. Describing the predicted cascades as genuine experimental mechanisms (Sec. 1) goes further than this evaluation supports.

The coherence scores are reported in Appendix Table S15 of the revised manuscript, computed by the grading LLMs under the original rubric (v1). Decoding and inference settings are reported in Appendix Section A.8 of the revised manuscript, specifically in Tables S13 and S14.

We agree that enrichment results are a proxy for regulator activity, and we have revised our claims in the paper to reflect this. We have additionally added a section on limitations in Appendix Section A.10 discussing this.

(continued in next comment)

Reviewer_Abu1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work targets the capacity of LLMs to explain biological treatment mechanisms. Given paired baseline and experimental conditions, the task requires the LLM to predict the whole pathway mechanism. The authors construct a dataset covering diverse biological domains, divided into training and test sets. Besides benchmarking current LLMs on the test set, they also train Qwen3.5 4B and observe effective improvement.

Critical strengths

The task is important and interesting. Pathway-based mechanism reasoning and phenotype prediction are key tasks in AI4bio. The query design in this work provides detailed baseline and perturbation conditions, which is a rational setting.

The work produces 15k samples, covering diverse biological domains, and supports both evaluation and training, making the scope of the work complete. The training is also effective according to the experiments.

Critical weaknesses

The reliability of the evaluation metric is questionable. The rubric judges the answer by its reasoning quality, which is easy to hack by producing "reasoning-like" content, but this is not guaranteed to indicate an improvement in correctness. The reference enrichment list also only includes items known to be potentially involved, but is not a valid indicator of the correctness of the result. I suggest that the paper should at least include a metric that only judges distinguishable content with known ground truth, such as predictions of known phenotypes. An example is the evaluation metric in [1].

Also, the evaluation based on Gemini 3.5 Flash-Lite is not reliable. It is necessary to assess the consistency of its scores with human annotations and those of a stronger model, such as GPT-6 Astra.

After training, the 4B model outperforms baselines including GPT 6 Luna, which means that the task is relatively easy. This is a major concern. I hope the authors can explain this.

[1] BioMaze: Benchmarking and Enhancing Large Language Models for Biological Pathway Reasoning. https://arxiv.org/pdf/2502.16660

Questions for authors

See the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Abu1 (Part 1/2)

Weakness 1: Reliability of the evaluation metric and evaluation against known biological evidence

The reliability of the evaluation metric is questionable. The rubric judges the answer by its reasoning quality, which is easy to hack by producing "reasoning-like" content, but this is not guaranteed to indicate an improvement in correctness. […] I suggest that the paper should at least include a metric that only judges distinguishable content with known ground truth, such as predictions of known phenotypes. An example is the evaluation metric in [1] (BioMaze).

Thank you for the thoughtful suggestions on evaluation. We agree with the reviewer that co-enrichment is not verified mechanistic ground truth; however, we regard enrichment computed on the samples as a useful proxy for the biological programs active in a bulk transcriptomic study. We note that the data-driven approach taken in TRACE serves as a strong complement to well-studied mechanistic literature-based evidence.

We have added BioMaze to our Related Work and acknowledge that literature-driven validated phenotypes can help in metric evaluation for biological reasoning models. We note that in our task setting, reasoning is focused on signaling cascades that distinguish bulk transcriptomic samples found in GEO studies. It is therefore difficult to find associated papers that have validated that exact signaling cascade. Moreover, the publications associated with a GEO study may not comprehensively describe the signaling pathways and enrichment terms identified through differential expression analysis of its transcriptomic samples. Thus, a BioMaze-style metric measuring against verified phenotypes is unavailable at scale for our setting. However, following the spirit of this suggestion, we have added two new metrics that do not use an LLM judge or the teacher.

(a) Metric 1: Frequency of disease-associated kinases and TFs supported by external evidence

The first metric we introduce is External Evidence Support (EES): for each answer, it counts the number of kinases and TFs supported by independent external evidence for that sample's disease, minus the number expected from other diseases. This adjusts for hub genes associated with many diseases. Answers naming no relevant entities score zero.

This metric can be calculated for different evidence sources. We use Open Targets [1] and DisGeNET [2] as our external evidence sources. For Open Targets, we use only non-literature evidence (genetic association, somatic mutation, known drug, and animal model).

Table R3: External Evidence Support (EES) on held-out GEO series. Named entities is the mean number of kinases and TFs named per answer. EES = 0 means no disease-specific support beyond hub genes. Values are mean ± SE.

ModelOpen Targets (n = 2,249): Named entitiesOpen Targets: EESDisGeNET (n = 1,608): Named entitiesDisGeNET: EES
Teacher reference4.30.28 ± 0.024.20.23 ± 0.02
Gemini 3.5 Flash-Lite1.10.11 ± 0.011.10.22 ± 0.01
GPT-6 Luna2.50.19 ± 0.022.40.35 ± 0.02
Global prior6.00.04 ± 0.026.00.06 ± 0.01
Disease prior5.90.21 ± 0.025.90.28 ± 0.02
kNN prior5.50.18 ± 0.025.50.24 ± 0.02
Gemini-kNN (retrieval)6.20.39 ± 0.036.30.54 ± 0.04
TRACE 4B (disease-pooled, ours)4.10.24 ± 0.034.10.26 ± 0.04
TRACE 4B (per-sample, ours)5.00.23 ± 0.025.10.45 ± 0.03

TRACE (per-sample) has a higher EES score on Open Targets and DisGeNET than all baseline models except Gemini-kNN, which represents a strong sample-retrieval baseline. This is to be expected, since Gemini-kNN provides Gemini with the most closely related samples and their enrichment at inference time. These results are added to the manuscript in Appendix Table S9.

References:

  1. Ghoussaini, Maya, et al. "Open Targets Genetics: systematic identification of trait-associated genes using large-scale genetics and functional genomics." Nucleic Acids Research 49.D1 (2021): D1311–D1320.
  2. Piñero, Janet, et al. "DisGeNET: a comprehensive platform integrating information on human disease-associated genes and variants." Nucleic Acids Research (2016): gkw943.

(continued in next comment)

AuthorsOpenReview ↗

Response to Reviewer Abu1 (Part 2/2)

(b) Metric 2: Gene Signature Sample Rank

The second metric we introduce is Gene Signature Sample Rank (GSSR): each answer's named kinases, TFs, and GO terms are expanded to their target or member genes using KEA, ChEA, and GO, giving the set of genes the hypothesis implicates. We score how well this gene set covers the sample's own differentially expressed genes (fraction covered, minus that expected for a random gene set of the same size), and compare it with the same score against every other signature of the same disease, excluding signatures from the same GEO study. The GSSR is the fraction of these other signatures that the sample's own signature outscores (0.0 = the answer fits its own sample worse than every other study of that disease; 0.5 = no sample-specific information, random chance; 1.0 = the answer fits its own sample better than every other study of the disease).

Table R4: Gene Signature Sample Rank (GSSR), paired comparisons. All methods are compared on the samples scored for every method (held-out GEO studies: n = 1,353 samples from 194 GEO series; held-out diseases: n = 190 samples from 115 GEO series). Values are mean ± SE.

ModelHeld-out GEO studiesHeld-out diseases
Teacher reference0.677 ± 0.0080.644 ± 0.021
Gemini 3.5 Flash-Lite0.511 ± 0.0080.518 ± 0.021
GPT-6 Luna0.519 ± 0.0080.489 ± 0.021
Global prior0.521 ± 0.0080.491 ± 0.022
Disease prior0.525 ± 0.008–
kNN prior0.534 ± 0.0080.538 ± 0.023
Gemini-kNN (retrieval)0.535 ± 0.0080.539 ± 0.023
TRACE 4B (disease-pooled, ours)0.536 ± 0.0080.501 ± 0.022
TRACE 4B (per-sample, ours)0.547 ± 0.0080.513 ± 0.022

On held-out GEO studies, TRACE variants have the highest score among methods without access to the closest training sample's enrichment evidence. TRACE (per-sample) in particular scores 0.547 on holdout GEO studies and 0.513 on holdout diseases. We note that although gains on this metric are small relative to the 0.5 chance score, sample ranking is done among samples enriched for the same diseases and disease-associated pathways, making them closely related and difficult to discriminate from one another. Any score above 0.5 therefore indicates an improvement in the model's ability to differentiate samples of the same disease.

Weakness 2/3: Reliability of Gemini grading, comparison with human annotations or stronger models, and task difficulty

The evaluation based on Gemini 3.5 Flash-Lite is not reliable. It is necessary to assess the consistency of its scores with human annotations and those of a stronger model, such as GPT-6 Astra.

After training, the 4B model outperforms baselines including GPT-6 Luna, which means that the task is relatively easy. This is a major concern.

Under the original rubric (v1), Gemini was the only grader and favored answers it had generated as the teacher. GPT-6 Astra was too expensive for the tens of thousands of grading calls involved; however, we use GPT-6 Luna, which is a lower-cost, high-performance option. Updated grading results with GPT-6 Luna and Claude Haiku 5.5 in addition to Gemini are provided in the revised manuscript and in the general comment addressed to all reviewers above. We base our main conclusions on the two non-teacher graders. Under rubric v2, the graders show strong agreement on model rankings, with Spearman correlations of 0.73–0.93. The judge-free metrics in Tables R3 and R4 above also provide complementary evaluation without relying on a grader LLM.

In the updated results, our 4B TRACE model still outperforms the GPT-6 Luna and Gemini LLM baselines. Frontier LLMs such as GPT-6 Luna and Gemini have much more advanced reasoning capabilities and general knowledge than TRACE 4B. What our results show is that the training data we are able to systematically generate teaches the smaller 4B model about enriched biological entities and involved pathways in disease contexts. This enables TRACE to perform better on our benchmark, which is focused on recovering data-driven enrichment terms and connecting them in coherent reasoning traces. We have explicitly stated this in our limitations section in Appendix A.10.

We are unable to obtain independent expert annotations during the rebuttal period for a sufficient number of benchmarking samples; however, we are working on obtaining independent expert evaluation of several randomly selected reasoning case studies from our TRACE 4B (disease-pooled) model. We will include these comments in our final submitted version, and if possible we will include them in an additional file within the rebuttal timeframe.

Hi reviewer Abu1,

Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers (Part 1/3)

We thank the reviewers for their constructive comments and careful assessment of our paper. We address shared concerns in this general response and reply to each reviewer's specific points below.

Since submission, we have made several major changes, reflected in the revised manuscript which has been uploaded on this OpenReview submission:

  • Independent graders and a revised rubric. We now report rubric scores from three graders from different model families (GPT, Gemini, and Claude), and we base our main conclusions on the two graders outside of the teacher model's family (Gemini).
  • Prior-based and retrieval baselines. We have added a global frequency prior, a per-disease prior, a nearest-neighbor (kNN) sample prior, and a strong Gemini-kNN prior where Gemini 3.5 Flash-Lite is given the three most similar training studies with their enrichment lists in context.
  • Additional non-LLM judge metrics. We have added two additional metrics to our experiments that do not depend on an LLM judge or the teacher LLM. These metrics assess (i) whether models mention enrichment terms supported by external evidence, and (ii) whether the model-predicted gene set, based on predicted enrichment terms, overlaps highly with ground truth differential gene signatures.
  • Per-sample enrichment reasoning dataset. To analyze per-sample enrichment rather than pooled-disease sample enrichment, we evaluated a second dataset where teacher reasoning traces are based on a per-sample enrichment, rather than consensus across multiple samples associated with a disease, as in the original dataset. We retrain TRACE models on this per-sample dataset, and denote them as TRACE (per-sample) in the additional results given in comments below as well as in the revised manuscript Appendix section. TRACE (disease-pooled) refers to the original model.
  • Additional holdout disease setting. We have trained an additional TRACE model with 40 diseases held out of training, compared with five in the original evaluation.

Independent graders

A common request was to add additional LLMs as judges for rubric-based evaluation, so that evaluation does not depend on the same model used to generate the teacher samples. The strongest frontier LLMs, such as GPT-6 Astra, were too expensive for the tens of thousands of grading calls required. We therefore chose a strong, cost-efficient grader from each of three model families: Gemini 3.5 Flash-Lite, GPT-6 Luna, and Claude Haiku 5.5.

Under the original rubric (v1), we found these graders disagreed substantially, even on the teacher's own answers:

ReferenceGemini 3.5 Flash-LiteGPT-6 LunaClaude Haiku 5.5Average
Teacher reference answers, given enrichment evidence10.00 ± 0.006.48 ± 0.045.48 ± 0.047.32 ± 0.02

The v1 rubric left too much to interpretation: 10 points were split into three categories, each scored up to three or four points. Gemini favored answers it had generated as the teacher, while Haiku and Luna deducted heavily for not recovering every enriched kinase and TF, or for not providing sufficient proof of mechanistic links in biological reasoning. Inspection of the grading rationales revealed inconsistent interpretations of coverage and evidence requirements in the original rubric (v1). We therefore rewrote the rubric as an explicit checklist with clearer grading criteria for enrichment-consistent hypothesis generation, which we detail and show results for below and in the revised manuscript.

(continued in next comment)

AuthorsOpenReview ↗

General Response to All Reviewers (Part 2/3)

Rubric v2

We made two changes and re-ran grading:

  • Checklist instead of analytic scoring. Points are now awarded based on 10 distinct criteria, rather than subjective scoring for categories.
  • Clearer criteria. Each item has explicit criteria for awarding its point. This reduces ambiguity for the grader, particularly for plausible mechanistic links consistent with the enrichment evidence and for crediting enrichment terms without requiring all of them to be present in the answer.

The full rubric is in the revised manuscript. The checklist items are:

ItemCriterion
G1The answer centers on at least one supported triplet: it names that triplet's kinase, transcription factor, and GO process. Close synonyms or gene-family members count.
G2It proposes a plausible relationship between a supported-triplet kinase and transcription factor, for example how the kinase could act on or feed into the TF. A brief proposed link is enough.
G3It connects the TF to the GO process through at least two of the triplet's overlapping genes, presented as targets or effectors of that process.
C1It uses at least one kinase and at least two transcription factors from the reference lists.
C2It addresses at least three distinct GO processes from the reference lists.
M1It proposes how the perturbed condition, relative to the control, engages the upstream signaling. A plausible hypothesized trigger is enough.
M2It presents an ordered cascade: upstream signaling → transcriptional regulation → affected genes or programs → biological process, rather than an unordered collection of entities.
M3It links the altered processes to the biological or disease context, or to a phenotype, through a stated functional consequence.
M4The reasoning trace and final answer agree on the core cascade: the same central kinase, TF, and process. Mark 0 if the reasoning trace is empty.
N1Entities are embedded in explanatory sentences. There are no long runs of names without stated roles or keyword dumps.

Under rubric v2, all three graders agree on high scores for the teacher reference answers:

Reference setGemini 3.5 Flash-LiteGPT-6 LunaClaude Haiku 5.5
Teacher reference answers, held-out GEO series9.94 ± 0.019.19 ± 0.029.02 ± 0.02
Teacher reference answers, held-out diseases9.97 ± 0.019.31 ± 0.059.16 ± 0.05
Teacher reference answers, 200 random training samples9.95 ± 0.029.20 ± 0.059.02 ± 0.05

Values are mean ± SE. Teacher reference answers are generated with access to enrichment evidence.

The graders also agree on how models rank against one another. The following table reports Spearman correlations of model rankings, including all evaluated baselines, SFT models, and teacher reference answers:

GraderGemini 3.5 Flash-LiteGPT-6 LunaClaude Haiku 5.5
Gemini 3.5 Flash-Lite1.000.730.85
GPT-6 Luna0.731.000.93
Claude Haiku 5.50.850.931.00

We base our main interpretation on the average of the two non-teacher graders, GPT-6 Luna and Claude Haiku 5.5, to reduce dependence on Gemini's preference for teacher-style answers.

(continued in next comment)

AuthorsOpenReview ↗

General Response to All Reviewers (Part 3/3)

Updated rubric results

Table R1: Rubric v2 scores (out of 10), held-out GEO series (n = 2,451).

ModelGemini 3.5 Flash-LiteGPT-6 LunaClaude Haiku 5.5Avg. non-teacherAvg. all
Teacher reference9.94 ± 0.019.19 ± 0.029.02 ± 0.029.10 ± 0.029.38 ± 0.01
Random shuffled reference3.03 ± 0.031.45 ± 0.021.39 ± 0.021.42 ± 0.021.96 ± 0.02
Qwen3.5-4B base5.02 ± 0.023.96 ± 0.034.11 ± 0.024.04 ± 0.024.36 ± 0.02
Gemma-3-4B-it4.86 ± 0.023.61 ± 0.033.62 ± 0.023.61 ± 0.024.03 ± 0.02
Llama-3.2-3B-Instruct4.64 ± 0.023.58 ± 0.033.48 ± 0.023.53 ± 0.023.90 ± 0.02
OpenBioLLM-Llama3-8B0.94 ± 0.030.64 ± 0.030.74 ± 0.030.69 ± 0.030.77 ± 0.03
DeepSeek-R1-Distill-Qwen-7B4.12 ± 0.023.13 ± 0.023.13 ± 0.023.13 ± 0.023.46 ± 0.02
Gemini 3.5 Flash-Lite5.82 ± 0.025.06 ± 0.025.31 ± 0.025.18 ± 0.025.39 ± 0.02
GPT-6 Luna5.33 ± 0.035.35 ± 0.025.44 ± 0.025.39 ± 0.025.37 ± 0.02
Gemini-kNN (retrieval)7.75 ± 0.037.04 ± 0.046.96 ± 0.037.00 ± 0.037.24 ± 0.03
TRACE 4B (disease-pooled, ours)9.43 ± 0.028.20 ± 0.038.15 ± 0.038.18 ± 0.038.60 ± 0.03
TRACE 4B (per-sample, ours)8.15 ± 0.046.34 ± 0.046.49 ± 0.046.40 ± 0.046.98 ± 0.04

Table R2: Rubric v2 scores (out of 10), original held-out diseases (n = 315, five diseases).

ModelGemini 3.5 Flash-LiteGPT-6 LunaClaude Haiku 5.5Avg. non-teacherAvg. all
Teacher reference9.97 ± 0.019.31 ± 0.059.16 ± 0.059.23 ± 0.049.48 ± 0.03
Random shuffled reference3.19 ± 0.081.47 ± 0.061.38 ± 0.061.43 ± 0.042.02 ± 0.04
Qwen3.5-4B base4.85 ± 0.053.51 ± 0.073.86 ± 0.063.69 ± 0.064.08 ± 0.05
Gemma-3-4B-it4.73 ± 0.062.97 ± 0.073.09 ± 0.053.03 ± 0.053.59 ± 0.05
Llama-3.2-3B-Instruct4.61 ± 0.063.19 ± 0.073.15 ± 0.053.18 ± 0.053.65 ± 0.05
OpenBioLLM-Llama3-8B1.17 ± 0.100.72 ± 0.070.82 ± 0.070.78 ± 0.070.91 ± 0.07
DeepSeek-R1-Distill-Qwen-7B4.14 ± 0.052.96 ± 0.072.98 ± 0.062.98 ± 0.063.37 ± 0.05
Gemini 3.5 Flash-Lite5.33 ± 0.054.68 ± 0.065.02 ± 0.044.85 ± 0.045.01 ± 0.04
GPT-6 Luna4.79 ± 0.084.85 ± 0.074.93 ± 0.064.90 ± 0.064.86 ± 0.06
Gemini-kNN (retrieval)7.15 ± 0.086.03 ± 0.116.24 ± 0.096.13 ± 0.106.48 ± 0.09
TRACE 4B (disease-pooled, ours)8.48 ± 0.086.93 ± 0.117.10 ± 0.107.02 ± 0.107.50 ± 0.09
TRACE 4B (per-sample, ours)7.40 ± 0.105.59 ± 0.115.76 ± 0.105.67 ± 0.106.25 ± 0.09

Values are mean ± SE over each model's graded rows; refusals and parse failures are excluded, and statistics on model completion rates and parsing failures are provided in Appendix Section A.8, Table S14 of the revised paper uploaded on OpenReview. Teacher reference: answers written by the teacher with access to the sample's enrichment evidence (upper reference). Random shuffled reference: negative control in which the teacher reference from a different sample is graded against the evaluated sample's evidence. Metrics are computed on samples successfully graded by every grader. We note that prior-only baselines output enrichment term lists rather than a natural language reasoning paragraph, so we compare these prior-only baselines on the two new metrics introduced below (evidence-supported enrichment terms and predicted gene set overlap ranking) rather than rubric grading.

Summary of the updated results

Under both non-teacher graders, TRACE (disease-pooled) scores over 2 points above GPT-6 Luna and Gemini 3.5 Flash-Lite on both held-out GEO series and the original held-out diseases. Gemini-kNN, a strong prior that augments Gemini 3.5 Flash-Lite with retrieval of nearest training samples in-context, scores closest to TRACE (disease-pooled) under the non-teacher average: 7.00 versus 8.18 on held-out GEO series and 6.13 versus 7.02 on held-out diseases. The TRACE (per-sample) model performs slightly worse than Gemini-kNN in both holdout settings; this is expected, since it was trained in a per-sample enrichment setting and Gemini-kNN represents a very strong prior baseline.

Finally, we want to reiterate to all reviewers that a central contribution of this work is the scalable enrichment and reasoning-trace generation pipeline we have developed to build the reasoning SFT and benchmark datasets. We anticipate that this pipeline will prove useful for generating data-driven biological reasoning training and benchmarking data for all LLMs. We remain committed to releasing this pipeline upon publication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.