acceptodds
Under review as a conference paper at ICLR 2027

From Measured Biology to Analysis Decisions: How Far Does Post-Trained Knowledge Travel?

Abstract

Closed-loop biological discovery requires models to do more than acquire domain knowledge: they must reuse prior experimental knowledge when analyzing new evidence. We study whether biological knowledge introduced through post-training can serve this role. We introduce MorphKnow, containing 15,004 experimentally measured phenotypic knowledge items derived from perturbational morphology, and MorphAnalysis, a paired suite of 967 data-analysis tasks that require combining prior biological knowledge with task-specific experimental evidence. We evaluate knowledge reuse across accessibility, inferential extension, and analytical application, and conduct controlled post-training experiments with supervised fine-tuning, distillation-based objectives, and reinforcement learning. We find a sharp separation between knowledge acquisition and downstream usability. Token-level post-training makes MorphKnow highly accessible and supports some unseen comparisons, but portability depends strongly on the operation required: it weakens for conditional lookups and held-out biological entities, and does not translate into improved scientific analysis. Supplying the correct facts does not restore analytical performance: models follow the biological knowledge but make less effective use of the current task evidence. Preserving analysis behavior during training, or explicitly prompting the model to recall the relevant fact before analysis, still does not produce reliable gains on the tasks that depend on that fact. These results suggest that parametric knowledge acquisition alone does not make biological knowledge a usable scientific prior, motivating post-training and evaluation that explicitly target knowledge-evidence integration.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
phenotypic drug discovery, post-training, scientific AI agents, biological knowledge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_opnSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether newly acquired biological knowledge through post-training can be transferred and effectively used in subsequent scientific analysis. The authors construct two complementary benchmarks, MorphKnow and MorphAnalysis, and evaluate representative post-training methods, including SFT, SD, OPD, and GRPO, from multiple perspectives such as knowledge accessibility, inferential extensibility, and analytical applicability. The results show that although existing methods can effectively acquire and memorize new biological knowledge, such knowledge does not necessarily transfer reliably to complex data-analysis tasks. Overall, the problem setting and benchmark construction are valuable, but some of the main findings are closely related to existing observations on catastrophic forgetting and knowledge transfer. In addition, the paper lacks a deeper mechanistic explanation of why acquired knowledge cannot be effectively used in downstream analysis, and there is still room for improvement in terms of model-scale analysis, training details, and experimental transparency.

Critical strengths
  1. The authors propose a new training dataset and a corresponding evaluation benchmark to systematically investigate a core question: when large language models acquire new biological knowledge through post-training, whether such knowledge can be effectively transferred and genuinely invoked and utilized in subsequent experimental data analysis.

  2. The authors systematically analyze representative post-training methods such as SFT, RLHF, and OPD from multiple evaluation dimensions, and reveal the limitations of these methods in knowledge transfer and downstream applications.

Critical weaknesses

To be honest, I do not have a research background in biology. Therefore, the following comments mainly evaluate the paper from the perspectives of its motivation, algorithmic aspects, and experimental methodology.

  1. From a purely algorithmic perspective, some of the conclusions presented in this paper have already been discussed in many previous works and are fairly common observations[1][2][3]. For example, Section 4.2 argues that acquiring knowledge does not necessarily mean that the knowledge can be effectively utilized in analytical tasks. Figure 2 also shows that models after SFT, SD, and OPD perform worse than the base model on analytical tasks. In fact, this observation is quite common. After the above training procedures, models are prone to catastrophic forgetting, resulting in degraded performance on the original tasks compared with the base model. In addition, it can be observed that GRPO effectively mitigates forgetting and maintains a similar capability to the base model, potentially due to its KL-divergence-based alignment. Similarly, in Section 6.1, the authors show that knowledge replay can prevent capability forgetting, which is also a commonly observed conclusion rather than a new insight.

  2. In addition, Section 4.2 presents the core conclusion that ``acquiring knowledge does not mean that the knowledge can be effectively utilized in analytical tasks.'' However, the paper does not provide a detailed analysis of why this happens. Why can the acquired knowledge not be effectively utilized? What causes this phenomenon? For example, is it caused by catastrophic forgetting after SFT? The authors mainly restate the experimental observations without providing the underlying reason that readers are most concerned about.

  3. In Section 5.1, the authors state that knowledge portability depends on the required operation, but they do not clearly explain what these operations refer to. For example, do these operations refer to algorithm-level techniques for preventing knowledge forgetting, such as knowledge replay, gradient clipping, or regularization? The authors need to further clarify what exactly is meant by these operations.

  4. In addition, from an experimental perspective, the authors only conduct experiments on 9B-scale models, lacking an analysis of knowledge transfer across models of different scales. The paper also lacks detailed demonstrations of the training process. For example, during GRPO training, how is the reward function designed for the biological domain? For SFT, how does the loss curve change with training steps? Similarly, for OPD, what are the parameter sizes of the teacher and student models? Is the teacher model trained with GRPO, or is the base model directly used as the teacher? These detailed experimental settings are critical for reproducing the core conclusions of the paper.

[1] Luo Y, Yang Z, Meng F, et al. An empirical study of catastrophic forgetting in large language models during continual fine-tuning[J]. IEEE Transactions on Audio, Speech and Language Processing, 2025.

[2] Li H, Ding L, Fang M, et al. Revisiting catastrophic forgetting in large language model tuning[C]//Findings of the association for computational linguistics: EMNLP 2024. 2024: 4297-4308.

[3] Abdi I, Gupta A, Mok M, et al. Evolutionary Strategies lead to Catastrophic Forgetting in LLMs[J]. arXiv preprint arXiv:2601.20861, 2026.

Questions for authors

See above weakness.

2 comments

As the discussion period is coming to a close, we would be grateful to know whether our response and the additional experiments have addressed your concerns, and we would be happy to clarify any remaining questions.

Dear Reviewer opnS,

The discussion period ends in a few hours, and we have not received a response to our rebuttal. Your main concerns are addressed by new results in the revised manuscript:

  • Catastrophic forgetting (W1): forgetting explains the drop below the Base model, not the missing gain. Once replay removes it, trained facts are recalled 35–38 points more accurately than held-out ones, yet the analyses that depend on them are no more accurate (−1.4 pp, pooled over two seeds).
  • Mechanism (W2, W3): with the analysis trajectory held identical up to the final decision, training on the fact shifts the probability of fact-consistent decisions by only +0.1 points; splitting IE-SP into two lookups raises accuracy from 0.690 to 1.000.
  • Model scale (W4): Qwen3.5-4B reproduces the main results of Sections 5.1 and 5.2.

Engaging with the authors' response is part of the review process, and we would appreciate the same level of care that went into our revision!

Reviewer_uS3JOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether LLMs can use biological knowledge learned during post-training in data analysis. The authors build MorphKnow with 15,004 facts from Cell Painting measurements and MorphAnalysis with 967 related analysis tasks. They find that the models can recall the facts with nearly perfect accuracy, but training on these facts gives little improvement on the analysis tasks, even with analysis replay.

Critical strengths
  1. The paired fact-split experiment (Sec. 6) is carefully controlled. It compares the same task with and without training on the supporting fact.
  2. The authors release the facts and analysis tasks based on real Cell Painting measurements, with reproducibility checks.
Critical weaknesses
  1. The main conclusion is too strong. SFT hurts the agent behavior even when the correct fact is given (Table 15). After replay, explicit recall improves AA-RD accuracy from about 0.56 to 0.65-0.69 (Fig. 10). This suggests that retrieval may be the bottleneck. The trained-vs-held-out comparison is also weakened by above-chance recall of held-out facts.
  2. The analysis tasks do not cleanly test the trained knowledge. AA-PS does not state that known biological relations should decide the answer, and most linked training facts are relative comparisons rather than the absolute statements in the oracle prompts. For AA-RD, the linked facts cover only 2-3 of the six readout blocks needed for the decision.
  3. The novelty is unclear. Prior work already studies the gap between recalling learned facts and using them, e.g. Allen-Zhu & Li (arXiv:2309.14402) and Balesni et al. (arXiv:2411.16353), but these are not cited.
  4. Each configuration uses only one training seed. GRPO also sees about 5K question instances compared with 150K for SFT, so the method comparison is not budget matched.
Questions for authors
  1. Can the authors use the existing runs in Fig. 10 to compare the gain from explicit recall against the base and replay-only models, with confidence intervals? How does AA-RD accuracy depend on whether the recalled fact is correct?
  2. Can the authors clarify how the AA-PS prompt tells the model to use known biological relations? I suggest testing an explicit criterion and training on the same statements used in the oracle prompts.
  3. Can the authors clarify the novelty relative to the prior work above? I also suggest adding training seeds and a budget-matched GRPO run.
2 comments

As the discussion period is coming to a close, we would be grateful to know whether our response and the additional experiments have addressed your concerns, and we would be happy to clarify any remaining questions.

Dear Reviewer uS3J,

The discussion period ends in a few hours, and we have not received a response to our rebuttal. Your main concerns are addressed by new results in the revised manuscript:

  • Recall gains (W1): the recall gain over Replay only is similar for trained and held-out facts (13.6 vs. 10.3 pp; the difference is not significant), so it is not a benefit of learning the particular fact.
  • Train–test matching (W2): neither stating the decision criterion nor training on the exact oracle statements closes the gap: the fact-specific effect is −0.4 and +5.2 pp, against gains of 41.6 and 55.5 pp when the same statements are in the prompt.
  • Novelty (W3): our work differs from prior studies in both what the model learns (relations measured in experiments, not synthetic facts) and what it must do with that knowledge (guide an analysis that also uses new evidence).
  • Seeds and GRPO budget (W4): a second seed gives a consistent result (−1.4 pp pooled, upper bound +0.2 pp), and GRPO trained for 200 steps, on more sequences than SFT, leaves the conclusions unchanged.

Engaging with the authors' response is part of the review process, and we would appreciate the same level of care that went into our revision!

Reviewer_uQu2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether experimentally measured biological knowledge acquired through post-training improves subsequent data analysis. The authors construct MorphKnow with 15,004 knowledge items and MorphAnalysis with 967 analysis tasks. They compare supervised fine-tuning, distillation, and reinforcement learning, primarily using Qwen3.5-9B. Knowledge answering improves substantially under paraphrasing, but analytical performance does not improve consistently. Supplied-fact conditions, training-trajectory analyses, and matched fact-allocation experiments are used to investigate this gap.

Critical strengths

The paired knowledge and analysis tasks address a concrete question. The matched fact-allocation experiment is particularly useful because it goes beyond comparing performance before and after training.

Critical weaknesses

1.Training-related ability changes remain an important alternative explanation. In Table 15, supplying the correct fact gives the base model 99.5% accuracy on readout selection, compared with 56.3% for SFT. Since both models receive the relevant fact, this difference cannot be explained simply by failure to recall knowledge. Replay training and the single-turn probes address parts of this concern, but the paper does not independently establish which relevant reasoning, numerical-analysis, and tool-execution skills have been preserved. The matched replay experiments are therefore central to the conclusion, and their remaining execution differences need to be quantified more clearly.

2.The statistical evidence supports a narrower conclusion than the overall narrative suggests. Appendix F uses one training seed per replay recipe. The upper bounds of +3.17 and +2.19 percentage points in Table 22 therefore do not describe variability across training runs or alternative fact allocations. Table 24 also reports a positive 5.5-point trained-versus-held-out advantage after explicit recall, whereas the additional improvement over no recall is an uncertain 2.8 points. These are different comparisons. The paper should specify which supports its claim of no reliable analytical benefit, rather than treating the interventions as having a single negative outcome.

3.The analytical task definition needs clarification. Section 2.4 states that the supporting fact alone determines the answer for Processing Selection, but Appendix B.6.3 says the model must inspect two matrices to identify which preserves the relation. Those are different tests. More broadly, the three controlled decision families provide a useful diagnostic setting, but do not establish how the findings carry over to ordinary analysis of experimental datasets.

4.The prompt specification is incomplete. Appendix B.6 describes inserting a reference-facts block without providing the complete condition-specific prompts. This matters because Appendix E.2.4 reports sensitivity to wording. Full prompts and tool definitions, or precise implementation references, are needed.

Questions for authors

1.Can you report matched before-and-after results on independent reasoning, numerical-analysis, and tool-execution tasks, including the replay-trained models, to establish which relevant abilities were preserved?

2.Which sources of variability do the confidence bounds in Table 22 cover? How much is known about sensitivity to the training seed and the allocation of facts?

3.How should the positive 5.5-point estimate in Table 24 be reconciled with the claim that explicit recall does not establish a fact-specific analytical benefit? Is the primary comparison the post-recall advantage or its change relative to no recall?

4.Does Processing Selection require both the learned relation and inspection of the candidate matrices? Which description governs the released tasks, and how does this affect the interpretation of failure?

5.Can you provide the complete prompts and tool definitions for the no-fact, supplied-fact, and recall-before-analysis conditions, including the wording changes examined in Appendix E.2.4?

2 comments

As the discussion period is coming to a close, we would be grateful to know whether our response and the additional experiments have addressed your concerns, and we would be happy to clarify any remaining questions.

Dear Reviewer uQu2,

The discussion period ends in a few hours, and we have not received a response to our rebuttal. Your main concerns are addressed by new results in the revised manuscript:

  • Ability loss versus knowledge use (W1): both replay schemes keep performance on the numerical-analysis and reasoning evaluations, which need no biological knowledge, at the Base-model level, so the missing fact-specific gain is not a consequence of the execution loss.
  • Seed sensitivity (W2): a second training seed gives a consistent result; pooled over the two seeds, the fact-specific effect is −1.4 pp, with a one-sided 95% upper bound of +0.2 pp.
  • Task design (W3): each task family is a decision that recurs in Cell Painting analysis, built from six public datasets, with only the numerical values generated within observed ranges. AA-PS requires reading the candidate correlation matrices with the tools and keeping the one that preserves the stated relations; strategies that ignore the facts are held at chance by construction (revised Section 2.4).
  • Prompts (W4): the complete prompts and tool definitions are in the new Appendix G.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their constructive feedback.

We would like to clarify three main contributions of our work. First, we introduce MorphKnow, 15,004 biological knowledge items measured in experiments, and MorphAnalysis, 967 analysis tasks in which this knowledge must be combined with new experimental evidence. Together they evaluate whether knowledge learned in post-training can guide later scientific analysis, a key requirement for closed-loop discovery. Second, we design a controlled experiment that tests whether learning a particular fact improves the analyses that depend on it, independently of general effects of training such as forgetting. Third, our experiments locate where knowledge use fails: the models store the facts reliably, but their reuse weakens for conditional lookups and held-out entities, and the learned facts bring no reliable gain in the analyses that depend on them. This suggests that post-training should target how learned knowledge is retrieved and combined with new evidence, not only whether it is acquired.

In response to the reviews, we added training seeds for the paired experiment, two evaluations that need no biological knowledge, recall and decision-step analyses, tests with matched training statements and an explicit decision criterion, Qwen3.5-4B results, longer GRPO training, and the complete prompts and tool definitions. We also clarified the task and operation definitions, the relation to catastrophic forgetting, training details, and related work. All revisions are highlighted in blue in the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.