- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a 4B-parameter LLM for study-level mechanistic hypothesis generation from biological condition descriptions. The also present a dataset of 15,251 transcriptomic contrasts, using enrichment-derived kinase–transcription factor–pathway cascades to generate supervised reasoning traces. Evaluation on held-out GEO studies and unseen diseases shows substantial improvements over open-source and commercial LLM baselines in recovering enriched regulatory entities.
The paper is well-written, clearly structured, and addresses an interesting problem with promising empirical results. However, it remains unclear whether the model learns genuine mechanistic reasoning or primarily captures associations between metadata and enrichment patterns.
Critical strengths
-
The authors benchmark TRACE against several open-source and commercial LLMs across held-out studies and unseen disease contexts, providing meaningful evidence of improvements on the proposed task. The evaluation also includes detailed biological case studies and analyses of individual regulatory entities (e.g., kinases, TFs, and pathways), going beyond aggregate performance metrics.
-
The authors construct a relatively large dataset of transcriptomic reasoning samples and introduce a systematic pipeline connecting differential-expression enrichment to mechanistic hypothesis generation. The integration of established biological resources and the proposed release of the dataset and pipeline constitute potentially valuable contributions for future research in transcriptomic interpretation and biological reasoning.
Critical weaknesses
-
The evaluation relies on a single LLM judge (Gemini 3.5 Flash-Lite), which is also used to generate the training reasoning traces. This raises concerns about potential evaluation bias and the robustness of the reported scores. It is unclear whether the relative model rankings would remain consistent under different, potentially stronger or independent evaluators. Comparing multiple LLM judges and reporting their agreement would substantially strengthen confidence in the results.
-
Although the paper provides detailed biological case studies, only two individual examples are examined in depth, and their selection criteria are not clearly specified. This raises concerns about how representative these examples are of the model's overall performance. More systematic validation, including randomly selected successful and unsuccessful predictions and independent expert assessment of the proposed regulatory cascades, would help establish whether the model generates biologically meaningful mechanisms rather than simply recovering enriched entities.
Questions for authors
-
What motivated the selection of Gemini 3.5 Flash-Lite as both the teacher model for reasoning-trace generation and the evaluator? Given the potential for teacher–evaluator bias, could the authors validate their results using independent LLM judges or expert biological assessments, and demonstrate that the relative model rankings remain consistent?
-
The paper presents two qualitative case studies (Brugada syndrome and hyperglycemia), but does not clarify how these examples were selected. Could the authors specify the selection criteria and provide additional randomly sampled examples, including unsuccessful predictions, to assess whether the illustrated mechanistic reasoning is representative of the model's overall performance?
3 comments
Hi reviewer 2nLi,
Thank you for your reviewing service. The authors have provided fairly extensive rebuttals. Please confirm that you have read them, state which of your concerns are or are not addressed, and whether that changes your recommendations.
I appreciate the author's response to my concerns and the performed experiments. I've increased my score.
Response to Reviewer 2nLi
Weakness 1 / Question 1: Teacher–evaluator bias and independent judges
We thank the reviewer for this feedback. We originally selected Gemini 3.5 Flash-Lite for both teacher generation and evaluation because it offered biological reasoning capabilities at a cost that allowed generating and grading tens of thousands of samples. We agree that this creates a risk of teacher–evaluator bias, and we address this concern in detail in the general comment addressed to all reviewers above, with multiple graders and a revised rubric. Under rubric v2, the three graders' model rankings are consistent (Spearman correlations of 0.73–0.93), and we base our main conclusions on the two graders outside the teacher's model family.
We also added two metrics independent of an LLM judge (Tables R3 and R4 in our response to Reviewer Abu1), providing complementary evidence of per-sample reasoning performance from our models.
Weakness 2 / Question 2: Case-study selection and representativeness
We thank the reviewer for this comment. The original two case studies were selected to illustrate reasoning on held-out diseases. We have added 20 randomly selected case studies in the PDF linked below, including both successful and unsuccessful predictions, so that the representativeness of the illustrated reasoning can be assessed.
Twenty reasoning traces: https://anonymous.4open.science/r/Anonymous_TRACE-DFDE/TRACE_20_supplementary_case_studies.pdf
We are working on obtaining comments from an expert biologist on these reasoning case studies, which we will include in the final submission and, time permitting, during this rebuttal period.