acceptodds
Under review as a conference paper at ICLR 2027

When Do Biological Reasoning Models Use Their Biological Inputs?

Abstract

Biological reasoning models use post-training to connect an LLMs to biological inputs, typically representations from a biological foundation model and biological text such as gene and pathway descriptions, functional annotations, and gene lists. Their benchmark accuracy is taken as evidence that the LLM reasons over their biological inputs. We test this assumption in six biological reasoning models across DNA, protein, and single cell tasks. We perturb one biological input while holding the query and other inputs fixed, construct evidence conflicts that pair the foundation model representation of one genome, protein, or cell with the text of another, fit linear probes to the representations the language model receives, and analyze reasoning traces against the biological inputs. We find that current post-training strategies do not ensure that foundation model representations contribute to task performance. Evo2 and ESM3 contribute little to BioReason and BioReason-Pro performance on the evaluated tasks. Shuffling the DNA sequence barely changes BioReason disease prediction accuracy, and in evidence conflicts the two models follow the text in 97.9% and 99.7% of cases. Linear probes trained on the Evo2 and ESM3 representations predict the task targets, so these foundation models encode information relevant to the task, but provide limited overall performance improvement to BioReason and BioReason-Pro. In contrast, foundation model inputs contribute to ChatNT, Prot2Text-V2, and CellWhisperer performance, and differentially expressed genes in the gene sentence contribute to Cell2Sentence-Scale performance. Across SFT and RL checkpoints of BioReason-Pro and 42 BioReason checkpoints, increases in accuracy do not imply greater performance contributions from biological inputs. In reasoning traces, BioReason traces misstate nucleotide changes, while BioReason-Pro traces describe functions omitted from final predictions under evidence conflicts. Future work should test whether post-training objectives that reward correct use of biological inputs improve their contribution to task performance.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
biological reasoning, multimodal learning, biological foundation models, modality reliance, post-training, reasoning traces

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.40 over 5 reviews, above 99% of the others.
Reviewer_YX1ROpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether biological reasoning models actually use the biological inputs provided to them, rather than relying primarily on textual annotations, task priors, or other correlated inputs. The authors evaluate six models spanning DNA, protein, and single-cell tasks using controlled input perturbations, evidence conflicts, linear probes, and analyses of generated reasoning traces. The results show that biological foundation-model representations do not always contribute substantially to downstream performance. In BioReason and BioReason-Pro, shuffling the DNA or protein representation has little effect on task accuracy, and the models follow conflicting text evidence in 97.9% and 99.7% of cases, respectively, even though linear probes can recover task-relevant information from the foundation-model representations. In contrast, biological inputs contribute more clearly to ChatNT, Prot2Text-V2, CellWhisperer, and Cell2Sentence-Scale. The paper also shows that higher benchmark accuracy after SFT or RL does not necessarily imply greater use of biological inputs, and that generated reasoning traces can be inconsistent with the supplied biological evidence. An auxiliary sequence-prediction objective increases sensitivity to DNA inputs, but its benefit is concentrated on genomes seen during training.

Critical strengths

The paper makes a valuable conceptual distinction between a representation encoding task-relevant information and a reasoning model actually using that representation for its prediction. The combination of perturbation tests, evidence conflicts, and linear probes is substantially more informative than reporting benchmark accuracy alone.

The evaluation spans DNA, protein, and single-cell models, and includes both task performance and reasoning-trace analyses. The finding that post-training accuracy can improve without increasing the contribution of a biological input is important for interpreting progress in biological reasoning systems.

Critical weaknesses

Some perturbation experiments rely on shuffled or resampled biological inputs, which may produce unrealistic out-of-distribution representations. A small change in performance under such a perturbation does not by itself prove that the model ignores the original biological input, since the model may react only to biologically plausible changes or may receive sufficient information from correlated inputs.

The evidence for input underuse is difficult to separate from redundancy among the supplied inputs. In BioReason and BioReason-Pro, textual annotations may already contain most of the information needed for the task, so the limited marginal contribution of Evo2 or ESM3 does not necessarily imply poor multimodal integration. More systematic controls that remove or weaken competing inputs would strengthen the causal interpretation.

The auxiliary-supervision experiment demonstrates increased sequence sensitivity, but the edited-base prediction result generalizes poorly to unseen genomes. Since 135 of 145 held-out queries contain genomes present in training, the large in-distribution gains should not be interpreted as reliable sequence reasoning without a stricter genome-held-out evaluation.

The reasoning-trace analysis is interesting but mainly descriptive. The relationship between trace faithfulness, final prediction correctness, and actual use of biological inputs remains unclear, especially for models whose traces are generated under a separate prompted rationale protocol rather than produced during the original evaluation.

Questions for authors

Could the authors include stronger in-distribution perturbation controls, such as biologically plausible sequence or protein substitutions matched for distributional statistics, to distinguish genuine input insensitivity from out-of-distribution effects caused by shuffling?

Can the authors evaluate BioReason and BioReason-Pro under conditions where the competing textual annotations are removed, masked, or carefully matched in informativeness? This would help determine whether the foundation-model representations are unused or simply redundant with text inputs.

Could the auxiliary sequence-supervision experiment be repeated with a strict genome-level split in which no genome from training appears in validation or test? The current overlap makes it difficult to assess whether the improved edited-base prediction reflects sequence reasoning or memorization.

For the reasoning-trace experiments, can the authors quantify whether claims in the trace causally affect the final answer, for example through trace interventions or answer evaluation with selected evidence statements removed? This would clarify the distinction between plausible rationales and explanations that reflect the model’s decision process.

Please consider discussing “PertMind: Eliciting Emergent Biological Reasoning in LLM via Reinforcement Learning on Cellular Perturbation Data.” This work is relevant to the paper’s analysis of post-training and biological reasoning from cellular perturbation data, and would help position the findings about accuracy gains versus actual use of biological inputs.

Please also consider discussing “Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning.” Its uncertainty-aware biomedical post-training and token-level supervision are relevant to the paper’s broader question of how post-training objectives can encourage models to use biologically meaningful information rather than relying on superficial or redundant cues.

2 comments
AuthorsOpenReview ↗

Response to Reviewer YX1R Part 1

We thank the Reviewer for their careful reading and for the additional experiments they proposed.

Response to YX1R.W1 and YX1R.Q1. We replaced the ESM3 and DNA inputs with those of a real protein or variant from another query. Unlike a shuffled sequence, each replacement is a natural protein/DNA variant, so its representation lies within the distribution of inputs that the foundation model and the projector receive during training, and the replacement changes the biological content of the input while preserving a realistic length, composition and sequence structure. For BioReason-Pro, we computed the ESM3 input from a real protein of a different InterPro family, with GO-GPT and InterPro unchanged, and Fmax⁡F_{\max} is 0.8554 on 14,102 proteins, compared with 0.8553 for the intact input. For BioReason, we took pairs of queries that have identical prompt text but different diseases and replaced the variant window of one query with the variant window of the other, keeping its reference window. Only the inserted variant then indicates which disease is correct. On 794 such queries, BioReason RL predicts the disease of the inserted variant in 46.1% and the disease of the original query in 47.6%, so it does not prefer the disease indicated by the variant. We have updated Section 4.1 to include the variant substitution and Section 5 to describe both substitutions. We have added the protein substitution to Appendix B.

Response to YX1R.W2 and YX1R.Q2. Fig. 2a,c evaluates both models with the competing text removed in steps. With the query gene masked and the other text removed, BioReason accuracy is 0.037 with intact DNA and 0.064 with shuffled DNA. Without GO-GPT and InterPro, BioReason-Pro reaches Fmax⁡F_{\max} 0.201 with the intact ESM3 input and 0.205 with the shuffled input. If the text were only redundant with ZEvo2\mathbf{Z}_{\mathrm{Evo2}} or ZESM3\mathbf{Z}_{\mathrm{ESM3}}, removing it would expose a contribution from the representation, and we do not observe one. A model trained with the text always present may not have learned to use the representation, so we also retrained BioReason with the text masked on half of the training steps. Under masked text, this model reaches 0.133 with intact DNA and 0.124 with shuffled DNA, so masking during training does not create a contribution from ZEvo2\mathbf{Z}_{\mathrm{Evo2}} either.

Response to YX1R.W3 and YX1R.Q3. We retrained both edited base supervision models on a split in which reference windows of the same gene that share any 50 bp are grouped, and each group is assigned to only one of training, validation and test. No genome in training therefore overlaps with a genome in validation or test. Over three seeds, edited base pair accuracy on the 146 test queries is 0.116 with 257-bp windows, below the 0.164 of the best constant base change chosen on the test queries. The improved edited base prediction on the original split therefore reflects recall of windows seen in training, and we have updated RQ4 to include this result as a direct test of Finding 4.

Response to YX1R.W4 and YX1R.Q4. We performed the trace interventions the Reviewer suggests on BioReason RL, whose traces are produced during the original evaluation. Replacing the gene named in the trace with the gene of another query, which has a different disease, changes the answer on 0.6% of 1,407 queries and never results in the disease of the substituted gene. Removing the sentences that name the gene results in an accuracy of 0.845, compared with 0.846 for the unedited trace. Replacing the disease named in the trace with another of the 37 diseases changes the answer on 49.7% of queries and lowers accuracy to 0.525. The disease stated in the trace therefore affects the answer, whereas the answer changes little when the sentences that reason from the gene to that disease are removed. The base change that the trace states does not depend on the DNA, since the RL checkpoint states T>C on all 145 held-out queries with intact and with shuffled DNA (Fig. 7a). We have updated RQ3 to distinguish what the traces state from what the answer depends on.

AuthorsOpenReview ↗

Response to Reviewer YX1R Part 2

Response to YX1R.Q5 and YX1R.Q6. We thank the Reviewer for these references and have added PertMind and Balanced Fine-Tuning to Section 2. PertMind trains a language model with reinforcement learning, using rewards computed from gene expression responses in a perturbation atlas, and reports transfer to tasks outside the post-training data. These rewards follow the principle of the post-training recipe in Appendix E.7, in which targets are determined by biological measurements. The input perturbations and evidence conflicts of our study complement the evaluation of such models alongside accuracy on new tasks. Balanced Fine-Tuning reweights tokens within groups and shifts weight towards knowledge-dense samples during supervised fine-tuning. Our RQ4 loss also weights tokens unequally, multiplying the loss on the tokens after a marker string by a factor of five. Token-level weighting of this kind provides a direct way to implement the recipe of Appendix E.7, by placing higher weight on the tokens determined by the biological input.

Reviewer_xxxPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks the question whether biological reasoning models benefit from the DNA, protein, and gene-expression data they receive. The authors change or remove parts of the input and check what happens to the answers. They also give the models biological data and text that disagree. Authors test what information is stored in the features that represent the biological input by small classifiers (probes). Other experiments examine written explanations, model size, further training, and an extra training task that asks the model to describe a DNA edit.

For some models, changing the biological input has little effect on the overall task score when other information is still available. Other models respond more strongly, including when important genes are removed. The probes can predict some labels from the internal features, while the models' written explanations sometimes contain errors or unsupported details. The paper shows that high accuracy alone does not establish how much each input helps, and that explanations can contain errors. Claims that the models ignore biological inputs need more careful wording.

Critical strengths
  1. Several controlled tests show when biological inputs help predictions. The authors combine sequence shuffling, input replacement, text removal, and gene removal. Different models respond differently, and matched controls help explain the changes. For example, removing selected cell-type-related genes hurts C2S much more than removing equally many genes with similar expression. Together, these tests provide useful evidence about input use beyond overall accuracy or fluent explanations.
Critical weaknesses
  1. The tests support narrower conclusions than some claims about input use. An unchanged overall score does not mean unchanged answers: BioReason RL changes 7.0% of answers, and BioReason-Pro changes 46.8% of its GO function-tag sets. Shuffling also preserves some sequence information, and following text in a conflict can be reasonable. Probe success alone does not show useful information beyond the other inputs. The DNA gain over text is small and mostly comes from PRNP. Explanation errors do not necessarily reveal what caused an answer. The extra training experiment also changes several factors together and shows little transfer to unseen genomes. These limits weaken stronger claims about why the models behave this way.

  2. Score differences and missing analysis details make key comparisons hard to judge. Protein probe AUROC uses confidence scores, while BioReason-Pro is scored using yes/no function claims. These allow different levels of ranking detail, so the gap is hard to interpret. Missing GO annotations also do not prove that a function is absent. CellWhisperer's rank score can fall even when label accuracy stays unchanged. Comparable scores and label-accuracy checks are needed. Some code, feature definitions, and predictions for the final combined-input probes are not present, limiting checks of the results and their uncertainty.

Questions for authors
  1. Does the protein probe advantage remain when probes and BioReason-Pro are evaluated with comparable scores, such as yes/no decisions for both? This would help separate better predictions from differences in how the outputs are scored.

  2. Under the same input changes, does CellWhisperer's label accuracy fall alongside its rank score? Please also report the gap between correct and competing label scores. This would clarify whether the rank drop reflects worse classification.

  3. How uncertain is the DNA probe's gain over text on the same examples, overall and without PRNP? Can you release the final Fig. 4 code, feature definitions, data groups, and predictions needed to check this comparison?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xxxP Part 1

We thank the Reviewer for their detailed and constructive review, which has helped us clarify several analyses. We agree with the Reviewer that an unchanged score does not mean unchanged answers, and we have revised our conclusion to state that BioReason and BioReason-Pro change their answers when a sequence representation changes, while these representations contribute little to performance in the evaluated settings. A linear probe recovers the relevant functional distinctions from the same representations, so the information is present but the models do not use it to improve their predictions.

Response to xxxP.W1.

Unchanged score and unchanged answers. The paper reports both (Tables S2 and S3). For BioReason, decoding the identical prompt twice changes 0 of 1,449 answers, so the 101 answers that change under shuffled DNA are caused by the change in ZEvo2\mathbf{Z}_{\mathrm{Evo2}}. These changes cancel in accuracy, with 36 incorrect answers becoming correct and 35 correct answers becoming incorrect. For BioReason-Pro, the GO term sets that change under shuffled ESM3 have no direction with respect to the annotations, since the per-protein F1F_1 improves for 11.2% of proteins and worsens for 10.8%. On 1,043 pairs of proteins of the same family with different functional labels, shuffled ESM3 changes the property prediction on four. Both models therefore respond to their sequence representation without translating it into better predictions.

Shuffling preserves sequence information. We computed the ESM3 input of BioReason-Pro from a real protein of a different InterPro family, with GO-GPT and InterPro unchanged, and Fmax⁡F_{\max} is 0.8554 compared with 0.8553 for the intact input on 14,102 proteins. For BioReason, we took pairs of queries that have identical prompt text but different diseases and replaced the variant window of one query with the variant window of the other, keeping its reference window. Only the inserted variant then indicates which disease is correct. On 794 such queries, BioReason RL predicts the disease of the inserted variant in 46.1% and the disease of the original query in 47.6%, so it does not prefer the disease indicated by the variant.

Following text in a conflict can be reasonable. We agree that following the text in a single conflict can be reasonable. Our conclusion is that BioReason-Pro follows the text in almost every conflict, whereas a model that uses its biological representation would follow the representation in some of them. In 106 conflicts between proteins of the same family with opposite labels, in which the protein without the function has an experimental GO NOT annotation, BioReason-Pro predicts the function supported by the GO-GPT and InterPro text in 98.1%.

Probe success and other inputs. For proteins, a probe on GO-GPT and ESM3 together scores higher than GO-GPT alone on all three properties (Section 4.1), and the families are selected so that the InterPro family does not determine the property. For DNA, we agree that the probe improves on the text-only predictor only on the PRNP queries, and we have updated Section 4.1 to report the PRNP queries and the other genes separately (see our response to xxxP.Q3).

Explanation errors and causes of the answer. We agree that an error in a trace does not show what caused the answer. To measure what the answer depends on, we edited the reasoning trace BioReason RL generated for each query and gave the model the prompt with the edited trace, so that it generated only the final answer. Replacing the gene named in the trace with the gene of another query, which is associated with a different disease label, changes the answer on 0.6% of 1,407 queries and never results in the disease of the substituted gene. Removing the sentences that name the gene results in an accuracy of 0.845, compared with 0.846 for the unedited trace. Replacing the disease named in the trace with another of the 37 diseases changes the answer on 49.7% of queries and lowers accuracy to 0.525. The disease stated in the trace therefore affects the answer, whereas the answer changes little when the sentences that reason from the gene to that disease are removed.

Training experiment and unseen genomes. The edited base target differs from the benchmark trace in both the bases it states and its length, so we trained a control whose target contains only the disease answer. With the benchmark trace, the answer-only target and the edited base target, shuffling the DNA lowers disease prediction accuracy by at most 0.02, within the variation across seeds. For unseen genomes, we retrained on a split in which no genome in training overlaps with a genome in validation or test. Over three seeds, edited base accuracy on the 146 test queries is 0.116, compared to 0.164 for the best constant base change. We have updated RQ4 to include this result.

AuthorsOpenReview ↗

Response to Reviewer xxxP Part 2

Response to xxxP.W2 and xxxP.Q1.

Table R1. Performance of BioReason-Pro and the ESM3 probe within InterPro families. We report the mean per-family AUROC on 540 proteins in 140 InterPro families, pooled over enzyme activity, organelle targeting and DNA binding, with a 95% confidence interval that resamples families. BioReason-Pro predicts a property when the property's GO term is in its output, so its score is binary and its per-family AUROC equals balanced accuracy. The probes are fitted on five folds grouped by family.

PredictorOutput scoredMean per-family AUROC [95% CI]
BioReason-Pro RLGO term present or absent0.830 [0.790, 0.869]
ESM3 probecontinuous probability0.868 [0.820, 0.912]
ESM3 probebinary, one threshold shared by all families, selected on validation families0.694 [0.663, 0.726]
ESM3 probebinary, threshold at a quantile of each family's scores, quantile selected on validation families0.853 [0.806, 0.897]

The probe advantage remains with binary scores for both, with a mean per-family AUROC of 0.853 [0.806, 0.897] compared to 0.830 [0.790, 0.869] for BioReason-Pro (Table R1). To make the probe scores binary, we rank the proteins of each family by probe score and assign the property to a fixed fraction at the top, with the fraction selected on validation families. The comparison tests the separation of proteins within InterPro families held out from probe fitting. The probe scores of a family share an offset relative to other families, so a threshold set within each family compares proteins of the same family. A threshold shared by all families separates whole families from each other, instead of the proteins within a family, and achieves a mean per-family AUROC of 0.694. BioReason-Pro’s property predictions have the same AUROC as the GO-GPT prediction in each of the 140 families. We have updated Appendix B to include the binary comparison beside the per-family ranking.

Missing GO annotations. We agree. For organelle targeting and DNA binding, the negative member lacks the GO annotation, which does not establish that the function is absent. The pseudoenzyme negatives have positive evidence of inactivity, since they are named "Inactive" in UniProt or listed as non-peptidase homologues in MEROPS. The GO NOT pairs provide stronger experimental evidence, with an annotation that the function is absent, and these pairs form the 106 conflicts reported above. We have stated this limitation where the 540 proteins are introduced in Section 4.1 and corrected the appendix description of the negatives.

Response to xxxP.W2 and xxxP.Q2.

Yes. We score each candidate cell type label by its summed log-likelihood given the CellWhisperer representation and take the label with the highest score. After counts are permuted across genes, label accuracy on 4,846 cells falls from 0.443 to 0.072, and it falls in every atlas. The score of the correct label also falls further below that of the highest-scoring other label, from −2.60 to −7.32 in log-likelihood, in every atlas. The decrease in the rank score therefore reflects worse classification. We have updated Appendix C to include these results.

AuthorsOpenReview ↗

Response to Reviewer xxxP Part 3

Response to xxxP.W2 and xxxP.Q3.

Table R2. Performance of BioReason and simple baselines on genome-dependent queries. We report disease prediction accuracy with a 95% confidence interval that resamples genomes together with all of their queries, separately for each row. A query is genome-dependent when queries with the same prompt text have more than one disease label in the benchmark. The text-only predictor and the Evo2 probe are fitted on folds grouped by genome, so the queries of a test genome are never in the training folds. BioReason was trained on the queries counted in the "BioReason trained on" column.

GeneQueriesGenomesBioReason trained onBioReason RL, intact DNA [95% CI]BioReason RL, shuffled DNA [95% CI]Text-only predictor [95% CI]Evo2 probe [95% CI]
PRNP9814760.714 [0.510, 0.898]0.714 [0.531, 0.878]0.708 [0.511, 0.878]0.855 [0.740, 0.944]
MET2412200.333 [0.125, 0.542]0.292 [0.083, 0.500]0.692 [0.567, 0.821]0.517 [0.358, 0.675]
VHL1414110.143 [0.000, 0.357]0.143 [0.000, 0.357]0.786 [0.571, 1.000]0.686 [0.457, 0.900]
PSAP1414130.857 [0.643, 1.000]0.857 [0.643, 1.000]0.857 [0.643, 1.000]0.693 [0.471, 0.893]
RET126110.333 [0.083, 0.667]0.333 [0.083, 0.667]0.492 [0.308, 0.633]0.333 [0.167, 0.500]
NRAS3300.333 [0.000, 1.000]0.333 [0.000, 1.000]0.667 [0.000, 1.000]0.667 [0.000, 1.000]
Non-PRNP6749550.403 [0.286, 0.537]0.388 [0.266, 0.522]0.709 [0.616, 0.800]0.563 [0.458, 0.670]
All165631310.588 [0.442, 0.712]0.582 [0.447, 0.695]0.708 [0.587, 0.817]0.736 [0.641, 0.816]

DNA probe uncertainty. On the 165 genome-dependent queries, the Evo2 probe reaches 0.736 [0.641, 0.816] and the text-only predictor, a majority vote that assigns the most frequent disease among training queries with the same prompt text, reaches 0.708 [0.587, 0.817] (Table R2). The probe exceeds the text-only predictor on the 98 PRNP queries, with 0.855 [0.740, 0.944] compared to 0.708 [0.511, 0.878], and not on the other 67, with 0.563 [0.458, 0.670] compared to 0.709 [0.616, 0.800]. The PRNP queries share one gene and two disease labels, while the other queries span five genes and ten disease labels, so the probe has few training queries for each of their diseases. BioReason RL reaches 0.714 on the PRNP queries compared to 0.855 for the probe. On the other genes, BioReason RL reaches 0.403 compared to 0.709 for the text-only predictor, despite being trained on 55 of those 67 queries. We have updated Section 4.1 to report the two groups of queries separately.

Code and predictions. We will add to the code release the Fig. 4 probe code, including the probe that combines text and Evo2 features. The release will also contain the feature definitions, the genome-grouped fold assignments, and the per-query predictions of every probe and of BioReason.

Thank you for the detailed response. The new experiments, address my main concerns and the narrowed claim is supported. I'm raising my score to 3 (weak accept).

Reviewer_4mMAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study analyzes bio-reasoning models in how much their success draws from the biological inputs. It benchmarks six such models covering genomic sequences, protein sequences, and cell transcriptome and answers four research questions with insights. The major findings were that some biological inputs contributed little to model performances, performance improvement in post training is not necessarily accompanied with more contributions from biological inputs, generated reasoning traces and rationales can misrepresent the biological inputs they describe, and auxiliary supervision to BioReason has no benefit in OOD performances.

Critical strengths
  • The study is probing the biological reasoning LLMs with input perturbations to understand and potentially improve their behavior.

  • The six models, the four tests, and the four research questions together present a comprehensive study.

Critical weaknesses
  • The findings are intuitively satisfying, more confirmatory than exploratory.

  • One critical question is whether the experimental protocols support the interpretations exactly, without mis- or over-interpretation. For instance, how sensitive were the conclusions to the way biological inputs were perturbed (shuffling to be specific, which maintain compositions), the fact that prompts also contain gene or protein information, and the permutation-invariant mean pooling used for probes.

  • Deeper insights connecting the model-specific findings with the models themselves, whether assumptions, architectural biases, or training, would strengthen the study with illuminating ideas for model improvement.

Questions for authors

Echoing the weaknesses mentioned above, could the authors please

  • provide more details and clarifications of the experiment protocols as the evidence for the interpretation fidelity; and
  • provide more insights connecting model behavior and model building?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4mMA

We thank the Reviewer for their thoughtful comments and for acknowledging our comprehensive study.

Response to 4mMA.W1. Several results were not expected from the published evaluations of these models. For example, BioReason reaches 0.858 accuracy when the DNA input is removed, compared with 0.847 with it. Post-training also behaves differently across models, since RL improves accuracy of BioReason and BioReason-Pro without raising the performance contribution of ZEvo2\mathbf{Z}_{\mathrm{Evo2}} or ZESM3\mathbf{Z}_{\mathrm{ESM3}}, while the larger C2S-Scale model shows a larger contribution from DEGs (Section 4.2). Edited base supervision makes the stated bases depend on the DNA, yet this does not transfer to genes absent from training. None of these outcomes can be inferred from the benchmark accuracies reported in the primary publications of these models.

Response to 4mMA.W2 and 4mMA.Q1. Shuffling permutes the bases of the DNA or the amino acids of the protein, so the shuffled sequence keeps the length and the base or amino acid composition of the original. We therefore also computed the ESM3 input of BioReason-Pro from the unmodified sequence of a different real protein, while the GO-GPT and InterPro text still describe the original protein. The replacement protein belongs to a different InterPro family, and we control for protein length and organism by selecting, where available, a replacement protein of similar length from the same organism. With the ESM3 input of the replacement protein, Fmax⁡F_{\max} is 0.8554 on 14,102 proteins, compared to 0.8553 with the intact input. The evidence conflicts of Fig. 3 also combine unmodified inputs from two queries.

The prompts also contain gene and protein text, which Fig. 2a,c removes in steps. With the gene and protein text removed, BioReason reaches 0.037 accuracy with intact DNA and 0.064 with shuffled DNA, and BioReason-Pro reaches Fmax⁡F_{\max} 0.201 with the intact ESM3 input and 0.205 with the shuffled input.

The probes of Section 4.1 are linear classifiers on the Evo2 or ESM3 output vectors averaged over all positions of the input. The average ignores the order of the output vectors, but each output vector depends on other positions of the input, so the average changes when the input order changes. On the 165 genome-dependent disease prediction queries, a probe on the Evo2 output vector at the edited base alone reaches 0.379 accuracy, compared to 0.738 for a probe on the average over the 2,048 positions of the variant window and 0.430 for always predicting the most common disease. The probe therefore predicts the disease from the whole window rather than from the edited base, and the average over positions results in higher accuracy than the vector at the edited base.

Appendix B describes how each perturbation and evidence conflict is constructed and lists the text fields present in each condition (Table S4). We have updated Appendix B to state the resampling unit of each interval. We have added the protein substitution and the edited base probe to Appendix B.

Response to 4mMA.W3 and 4mMA.Q2. BioReason and BioReason-Pro make little use of their biological input and are the only evaluated models whose training prompts always contain text that predicts the answer (Appendix B.5). BioReason's disease prediction did not depend on the DNA after we retrained it with the text masked on half of the training steps, with accuracy under masked text of 0.133 for intact DNA and 0.124 for shuffled DNA. In RQ4, the bases stated by BioReason depended on the DNA after edited base supervision, but only for genes seen in training. This suggests a training recipe where targets are determined only by the biological input and models are evaluated on genes held out from training, which we now discuss in Section 5.

The responses are helpful and appreciated. They do not materially change how the current results are interpreted or evaluated. But I would like to explain why below so that follow-up discussions can further clarify the subject and the contributions.

  • The new mismatch perturbation is useful addition to shuffling. The question remains in what kind of hard negatives in pertinent biological contexts should be constructed for a more biologically meaningful test on current models. For instance, for protein function prediction, instead of matching random pairs of protein sequences and InterPro text, what if one picks a different sequence in the same InterPro family that has a different set of GO terms?
  • I wonder whether I am following the gene/protein text ablation results correctly. They seem to reinforce my conjecture that most of the gene or protein information utilized by BioReason to perform the tested tasks was actually from the gene/protein text, not necessarily their sequence. This is not necessarily an issue, considering the rich semantic meaning and knowledge that can be learned about a gene or protein through natural texts, even just names. If that is the case, it can be misleading to claim that BioReason did not change its performance significantly when the gene/protein input is changed or imply that it seems to make little use of biological inputs. The biological inputs here for BioReason include both sequence and natural text. It is just that the latter is relied upon more?
  • Based on the bullet point above, masking texts and emphasizing more sequences in training may not be appealing as they first look, because the real question is how to achieve the synergy among various modalities of the biological input, rather than focusing on some part of the modalities (even though ESM3 embeddings include sequence, structure, function, and evolution information)?
  • The comment about permutation-invariance was about the probe, not the language model embedding.
AuthorsOpenReview ↗

Response to 4mMA second comment

We thank the Reviewer for the follow-up and we have updated the manuscript accordingly.

Response to point 1. Closely related proteins with different functions provide a more informative test than an unrelated replacement sequence. Our previous response emphasized the replacement with a protein from another family, but Figure 3b and Appendix B.2 already include pairs of proteins from the same organism and InterPro family that differ in one GO functional property, such as enzyme or pseudoenzyme activity, organelle targeting, or DNA binding. We view these questions as hard negatives for BioReason-Pro. We replace the ESM3 representation with that of the other protein and keep the original GO-GPT and InterPro inputs. On 378 conflicts from 189 pairs that BioReason-Pro answers correctly on the intact input, its predictions are consistent with the GO-GPT and InterPro inputs in 99.7% of cases. On the pairs where one protein has an experimental GO NOT annotation, which provides evidence that the function is absent, predictions are consistent with the text in 98.1% of 106 conflicts.

Response to point 2. We agree with the Reviewer that biological text, such as gene names and functional annotations, is a biological input that the models use. Our claims of a limited performance contribution refer to the Evo2 and ESM3 representations, which we test on queries where the text alone does not provide the answer. For BioReason, the 165 genome-dependent queries have prompt text that occurs with more than one disease label. On the 98 PRNP queries, BioReason RL reaches 0.714 accuracy compared to 0.855 for a linear probe on the Evo2 representation, and on the 67 queries of the other genes it reaches 0.403 compared to 0.709 for a majority vote over training queries with the same prompt text. For BioReason-Pro, proteins of the same InterPro family that differ in enzyme activity, organelle targeting, or DNA binding share the same InterPro annotation. When the ESM3 representation comes from the other protein of such a pair, BioReason-Pro predictions are consistent with the GO-GPT and InterPro inputs in 99.7% of 378 conflicts. We have updated the Introduction to state that text annotations can support accurate answers without the model using the sequence or transcriptomic representations, and the Conclusion to clarify that some biological inputs, such as foundation model representations, contribute little to performance.

Response to point 3. The goal is to combine complementary information from the text and the biological representations, not to increase reliance on the sequence. After we retrain BioReason with the text masked on half of the training steps, accuracy on 145 held-out queries with all text is 0.908 with intact DNA and 0.880 with shuffled DNA. Edited base supervision makes the stated bases depend on the DNA, but shuffling the DNA lowers disease prediction accuracy by 0.021 compared to 0.549 for the edited base pair. Neither change is sufficient on its own. We have updated the Discussion to state that training also needs queries with the same text but different answers, so that the model learns to use the biological representation to separate the answers.

Response to point 4. We appreciate the clarification from the reviewer on permutation-invariance. The linear probes use representations mean-pooled over positions, which makes them invariant to position, while the reasoning models receive a projected representation at each position. The probes show that task targets are predictable from the pooled representations but do not show whether the language model uses position-specific information. We have updated the Discussion to state this limitation.

Reviewer_kaG2OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates a problem regarding biological reasoning models. They are large language models that typically receive two types of biological input: a representation of genome, protein or a cell produced by a biological foundation model like Evo2; and biological text like gene names and pathway description. Specifically the problem the paper tackles is that the benchmark accuracy of these biological reasoning models is commonly taken as evidence that these models reason over their foundation model representations, while the paper tests whether this assumption is actually true or whether the models instead rely mainly on the biological text.

6 models were tested with 4 methods. Method 1 shuffle: scramble the DNA, protein, or cell input, then check whether the accuracy drops Method 2 conflict: give protein A’s embedding with protein B’s text, then see which one the answer follows. Method 3 probe: check whether the embedding contains the answer at all. Method 4 reasoning: check whether the model’s explanation matches its input.

Several main findings are reported including: 1. Evo2 and ESM3 embeddings contribute little to BioReason and BioReason-Pro performance. 2. The other four models do use their biology inputs. 3. Post-training (SFT and RL) raises BioReason and BioReason-Pro accuracy without increasing the contribution of their embeddings. 4. The explanations can be wrong. BioReason misstates nucleotide changes. 5. An auxiliary edited-base task makes BioReason's base predictions depend on the DNA, but shuffling the DNA still barely changes disease prediction accuracy, and the edited-base gain is concentrated in genomes seen during training. etc.

Critical strengths

Intuitive and important idea, clear and easy-to-follow writing, careful and honest experiment design and results presentation, thorough discussion of existing related work. Applying the method to biology reasoning model is new as far as I can tell. The insights and findings are useful.

Critical weaknesses

I don't find any obvious weakness. The paper is sound and well-supported in general. The list of models could be expanded a little bit to make the results more convincing but current scale is also acceptable to me

Questions for authors

On the 165 genome-dependent queries, the Evo2 probe beats the text-only predictor by about 0.03 (0.736 vs. 0.708), mostly on PRNP queries (98 of 165). Does this advantage remain when the PRNP queries are excluded? The paper reports PRNP accuracy (0.855 vs. 0.708, p.6). Please report probe and text-only accuracy for the 67 non-PRNP queries, and confirm whether the 0.708 on p.6 is computed on PRNP queries only, since it equals the 165-query value.

1 comment
AuthorsOpenReview ↗

Response to Reviewer kaG2

We thank the Reviewer for their encouraging review.

Response to kaG2.W1. In addition to BioReason, BioReason-Pro and C2S-Scale, the paper evaluates ChatNT, Prot2Text-V2 and CellWhisperer as positive controls, which receive DNA, protein and transcriptome representations through a projector. For CellWhisperer, we added analyses of two chat checkpoints in the rebuttal (see our response to KEA9.W1 and KEA9.Q1). The performance of these positive controls decreases under the same perturbations, so the evaluation detects use of the biological input when it occurs.

Response to kaG2.Q1. The Evo2 probe exceeds the text-only predictor on the PRNP queries and not on the other genes, while BioReason RL is below the probe on the PRNP queries and below the text-only predictor on the other genes. On the 98 PRNP queries, the probe reaches 0.855 accuracy compared to 0.708 for the text-only predictor and 0.714 for BioReason RL. These queries share one gene and two disease labels, so the probe has many training queries for each disease. On the 67 queries of the other five genes, the text-only predictor assigns the most frequent training disease for the same prompt text and reaches 0.709, compared to 0.563 for the probe and 0.403 for BioReason RL. These queries span ten disease labels, so the probe has few training queries for each disease. The text-only predictor reaches 0.7082 on the PRNP queries and 0.7085 on all 165 queries, so 0.708 appeared twice on p.6, and we removed the PRNP value from Section 4.1. We have updated Section 4.1 to include the values for the other five genes and for BioReason RL, and Appendix B to include both groups of queries.

Reviewer_KEA9OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the high benchmark accuracy of biological reasoning models genuinely stems from reasoning over their biological inputs (continuous foundation model representations and biological text). Across six models spanning three modalities — DNA, protein, and single-cell- the authors design several core diagnostic tests.

Critical strengths

This is an interesting and timely paper. Multimodal biological models are now numerous, but whether they actually utilize the information in their inputs — a question extensively explored in mainstream AI — has not been systematically investigated.

The writing is clear and easy to follow, and the manuscript is well-organized.

The benchmark yields several informative findings: for instance, models differ substantially in how much they rely on their biological inputs, to the point where perturbing an input leaves performance largely unchanged.

The modality coverage is broad, spanning the three core domains of genomics, proteins, and single-cell transcriptomics.

Critical weaknesses

C2S-Scale is essentially a text-only model, lacking both a continuous foundation model representation ZreprZ_{\text{repr}} and a modality projector.

Appendix B.5 identifies random masking of text fields during training (modality dropout) as the key factor determining whether a model utilizes its biological representations; however, this central causal hypothesis — training-time text masking — is never directly verified experimentally.

The paper would be strengthened if the authors proposed and discussed a constructive training scheme that genuinely mitigates the text shortcut and improves the utilization of continuous biological representations.

Questions for authors

Could the authors include evaluations of models that incorporate a continuous transcriptomic representation and a projector, specifically under generative reasoning traces (RQ3) and across training configurations (RQ2)?

Could the authors provide direct experimental verification of training-time random text masking (modality dropout)?

Could the authors discuss, as part of the outlook, a constructive post-training scheme to improve the utilization of continuous biological representations?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KEA9

We thank the Reviewer for their positive assessment and for the suggestions on further discussion of training improvements.

Response to KEA9.W1 and KEA9.Q1. We agree that C2S-Scale receives the transcriptome as text, which is why the paper also evaluates CellWhisperer, a model that encodes the transcriptome with Geneformer and passes the representation through a projector to the language model. In RQ1, permuting counts across genes lowers CellWhisperer's $1-$perplexity quantile in all five atlases (Fig. 2f), and in evidence conflicts CellWhisperer predicts the cell type supported by the representation in 96.1% of conflicts (Fig. 3e).

We now also evaluate CellWhisperer under RQ2 and RQ3. For RQ2, CellWhisperer's authors release two chat checkpoints that share an architecture and differ in their fine-tuning conversations. Permuting counts lowers label accuracy from 0.443 to 0.072 for the checkpoint fine-tuned on general conversations and from 0.384 to 0.076 for the checkpoint fine-tuned on cell type questions, so the transcriptome representation contributes to both checkpoints (Appendix C). For RQ3, we asked CellWhisperer to name the marker genes of a cell from the representation alone. Of the first five genes it names, 59.3% are DEGs of the annotated cell type, compared to 19.2% for genes drawn at random from the same cells, and permuting counts lowers this precision to 1.8% (Appendix D). Under the same analysis, C2S-Scale 27B reaches 21.6%, compared to 16.1% at random. CellWhisperer therefore uses its continuous transcriptome representation in its answers and in its generated text.

Response to KEA9.W2 and KEA9.Q2. We agree that Appendix B.5 does not test text masking directly. Among its six models, BioReason and BioReason-Pro are trained with prompts that always contain text that predicts the answer, and their performance is nearly unchanged under sequence shuffling. Both models also differ from the other four in architecture and scale, so we retrained BioReason with the text masked on half of the training steps, removing the same fields as the most restricted text condition of Fig. 2a. With masked text at evaluation, the retrained model reaches 0.133 accuracy with intact DNA and 0.124 with shuffled DNA on the 145 held-out queries, compared to 0.090 for always answering the most frequent training disease. Masking during training therefore teaches the model to answer within the label set when the text is absent, but its answers do not depend on the DNA. We have updated RQ4 to include this experiment, with its protocol in Appendix E.

Response to KEA9.W3 and KEA9.Q3. We have extended the outlook in Section 5 with what our experiments show about training. BioReason's disease prediction did not depend on the DNA after we retrained it with the text masked on half of the training steps. Prot2Text-V2 describes the protein supported by the sequence in 97.0% of evidence conflicts, and its training replaces the protein name and taxon in 80% of steps. In RQ4, the bases stated by BioReason depended on the DNA after edited base supervision, but only for genes seen in training.

To improve the utilization of continuous biological representations we propose a post-training recipe in which the supervision targets depend on the biological input. During SFT, the recipe applies modality dropout to the text inputs at a high rate and replaces the target of each dropped step with an auxiliary target determined by the remaining input, such as the edited bases. The recipe augments the training set with counterfactual pairs of queries that share the same text but have different labels, such as genome-dependent BioReason queries and variant substitutions described in Section 4.1, in which biological input alone distinguishes the correct answers. During RL, the recipe should compute a paired reward over each query and its counterfactual partner, so a policy obtains the full reward only when its answers follow the change in the biological input. We describe this recipe in Appendix E.7 as a direction for future work.

Thank you for your detailed response and additional clarifications. I carefully read your rebuttal and the other reviews. Therefore, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.