- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether biological reasoning models actually use the biological inputs provided to them, rather than relying primarily on textual annotations, task priors, or other correlated inputs. The authors evaluate six models spanning DNA, protein, and single-cell tasks using controlled input perturbations, evidence conflicts, linear probes, and analyses of generated reasoning traces. The results show that biological foundation-model representations do not always contribute substantially to downstream performance. In BioReason and BioReason-Pro, shuffling the DNA or protein representation has little effect on task accuracy, and the models follow conflicting text evidence in 97.9% and 99.7% of cases, respectively, even though linear probes can recover task-relevant information from the foundation-model representations. In contrast, biological inputs contribute more clearly to ChatNT, Prot2Text-V2, CellWhisperer, and Cell2Sentence-Scale. The paper also shows that higher benchmark accuracy after SFT or RL does not necessarily imply greater use of biological inputs, and that generated reasoning traces can be inconsistent with the supplied biological evidence. An auxiliary sequence-prediction objective increases sensitivity to DNA inputs, but its benefit is concentrated on genomes seen during training.
Critical strengths
The paper makes a valuable conceptual distinction between a representation encoding task-relevant information and a reasoning model actually using that representation for its prediction. The combination of perturbation tests, evidence conflicts, and linear probes is substantially more informative than reporting benchmark accuracy alone.
The evaluation spans DNA, protein, and single-cell models, and includes both task performance and reasoning-trace analyses. The finding that post-training accuracy can improve without increasing the contribution of a biological input is important for interpreting progress in biological reasoning systems.
Critical weaknesses
Some perturbation experiments rely on shuffled or resampled biological inputs, which may produce unrealistic out-of-distribution representations. A small change in performance under such a perturbation does not by itself prove that the model ignores the original biological input, since the model may react only to biologically plausible changes or may receive sufficient information from correlated inputs.
The evidence for input underuse is difficult to separate from redundancy among the supplied inputs. In BioReason and BioReason-Pro, textual annotations may already contain most of the information needed for the task, so the limited marginal contribution of Evo2 or ESM3 does not necessarily imply poor multimodal integration. More systematic controls that remove or weaken competing inputs would strengthen the causal interpretation.
The auxiliary-supervision experiment demonstrates increased sequence sensitivity, but the edited-base prediction result generalizes poorly to unseen genomes. Since 135 of 145 held-out queries contain genomes present in training, the large in-distribution gains should not be interpreted as reliable sequence reasoning without a stricter genome-held-out evaluation.
The reasoning-trace analysis is interesting but mainly descriptive. The relationship between trace faithfulness, final prediction correctness, and actual use of biological inputs remains unclear, especially for models whose traces are generated under a separate prompted rationale protocol rather than produced during the original evaluation.
Questions for authors
Could the authors include stronger in-distribution perturbation controls, such as biologically plausible sequence or protein substitutions matched for distributional statistics, to distinguish genuine input insensitivity from out-of-distribution effects caused by shuffling?
Can the authors evaluate BioReason and BioReason-Pro under conditions where the competing textual annotations are removed, masked, or carefully matched in informativeness? This would help determine whether the foundation-model representations are unused or simply redundant with text inputs.
Could the auxiliary sequence-supervision experiment be repeated with a strict genome-level split in which no genome from training appears in validation or test? The current overlap makes it difficult to assess whether the improved edited-base prediction reflects sequence reasoning or memorization.
For the reasoning-trace experiments, can the authors quantify whether claims in the trace causally affect the final answer, for example through trace interventions or answer evaluation with selected evidence statements removed? This would clarify the distinction between plausible rationales and explanations that reflect the model’s decision process.
Please consider discussing “PertMind: Eliciting Emergent Biological Reasoning in LLM via Reinforcement Learning on Cellular Perturbation Data.” This work is relevant to the paper’s analysis of post-training and biological reasoning from cellular perturbation data, and would help position the findings about accuracy gains versus actual use of biological inputs.
Please also consider discussing “Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning.” Its uncertainty-aware biomedical post-training and token-level supervision are relevant to the paper’s broader question of how post-training objectives can encourage models to use biologically meaningful information rather than relying on superficial or redundant cues.
2 comments
Response to Reviewer YX1R Part 2
Response to YX1R.Q5 and YX1R.Q6. We thank the Reviewer for these references and have added PertMind and Balanced Fine-Tuning to Section 2. PertMind trains a language model with reinforcement learning, using rewards computed from gene expression responses in a perturbation atlas, and reports transfer to tasks outside the post-training data. These rewards follow the principle of the post-training recipe in Appendix E.7, in which targets are determined by biological measurements. The input perturbations and evidence conflicts of our study complement the evaluation of such models alongside accuracy on new tasks. Balanced Fine-Tuning reweights tokens within groups and shifts weight towards knowledge-dense samples during supervised fine-tuning. Our RQ4 loss also weights tokens unequally, multiplying the loss on the tokens after a marker string by a factor of five. Token-level weighting of this kind provides a direct way to implement the recipe of Appendix E.7, by placing higher weight on the tokens determined by the biological input.
Response to Reviewer YX1R Part 1
We thank the Reviewer for their careful reading and for the additional experiments they proposed.
Response to YX1R.W1 and YX1R.Q1. We replaced the ESM3 and DNA inputs with those of a real protein or variant from another query. Unlike a shuffled sequence, each replacement is a natural protein/DNA variant, so its representation lies within the distribution of inputs that the foundation model and the projector receive during training, and the replacement changes the biological content of the input while preserving a realistic length, composition and sequence structure. For BioReason-Pro, we computed the ESM3 input from a real protein of a different InterPro family, with GO-GPT and InterPro unchanged, and Fmax is 0.8554 on 14,102 proteins, compared with 0.8553 for the intact input. For BioReason, we took pairs of queries that have identical prompt text but different diseases and replaced the variant window of one query with the variant window of the other, keeping its reference window. Only the inserted variant then indicates which disease is correct. On 794 such queries, BioReason RL predicts the disease of the inserted variant in 46.1% and the disease of the original query in 47.6%, so it does not prefer the disease indicated by the variant. We have updated Section 4.1 to include the variant substitution and Section 5 to describe both substitutions. We have added the protein substitution to Appendix B.
Response to YX1R.W2 and YX1R.Q2. Fig. 2a,c evaluates both models with the competing text removed in steps. With the query gene masked and the other text removed, BioReason accuracy is 0.037 with intact DNA and 0.064 with shuffled DNA. Without GO-GPT and InterPro, BioReason-Pro reaches Fmax 0.201 with the intact ESM3 input and 0.205 with the shuffled input. If the text were only redundant with ZEvo2 or ZESM3, removing it would expose a contribution from the representation, and we do not observe one. A model trained with the text always present may not have learned to use the representation, so we also retrained BioReason with the text masked on half of the training steps. Under masked text, this model reaches 0.133 with intact DNA and 0.124 with shuffled DNA, so masking during training does not create a contribution from ZEvo2 either.
Response to YX1R.W3 and YX1R.Q3. We retrained both edited base supervision models on a split in which reference windows of the same gene that share any 50 bp are grouped, and each group is assigned to only one of training, validation and test. No genome in training therefore overlaps with a genome in validation or test. Over three seeds, edited base pair accuracy on the 146 test queries is 0.116 with 257-bp windows, below the 0.164 of the best constant base change chosen on the test queries. The improved edited base prediction on the original split therefore reflects recall of windows seen in training, and we have updated RQ4 to include this result as a direct test of Finding 4.
Response to YX1R.W4 and YX1R.Q4. We performed the trace interventions the Reviewer suggests on BioReason RL, whose traces are produced during the original evaluation. Replacing the gene named in the trace with the gene of another query, which has a different disease, changes the answer on 0.6% of 1,407 queries and never results in the disease of the substituted gene. Removing the sentences that name the gene results in an accuracy of 0.845, compared with 0.846 for the unedited trace. Replacing the disease named in the trace with another of the 37 diseases changes the answer on 49.7% of queries and lowers accuracy to 0.525. The disease stated in the trace therefore affects the answer, whereas the answer changes little when the sentences that reason from the gene to that disease are removed. The base change that the trace states does not depend on the DNA, since the RL checkpoint states T>C on all 145 held-out queries with intact and with shuffled DNA (Fig. 7a). We have updated RQ3 to distinguish what the traces state from what the answer depends on.