- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether newly acquired biological knowledge through post-training can be transferred and effectively used in subsequent scientific analysis. The authors construct two complementary benchmarks, MorphKnow and MorphAnalysis, and evaluate representative post-training methods, including SFT, SD, OPD, and GRPO, from multiple perspectives such as knowledge accessibility, inferential extensibility, and analytical applicability. The results show that although existing methods can effectively acquire and memorize new biological knowledge, such knowledge does not necessarily transfer reliably to complex data-analysis tasks. Overall, the problem setting and benchmark construction are valuable, but some of the main findings are closely related to existing observations on catastrophic forgetting and knowledge transfer. In addition, the paper lacks a deeper mechanistic explanation of why acquired knowledge cannot be effectively used in downstream analysis, and there is still room for improvement in terms of model-scale analysis, training details, and experimental transparency.
Critical strengths
-
The authors propose a new training dataset and a corresponding evaluation benchmark to systematically investigate a core question: when large language models acquire new biological knowledge through post-training, whether such knowledge can be effectively transferred and genuinely invoked and utilized in subsequent experimental data analysis.
-
The authors systematically analyze representative post-training methods such as SFT, RLHF, and OPD from multiple evaluation dimensions, and reveal the limitations of these methods in knowledge transfer and downstream applications.
Critical weaknesses
To be honest, I do not have a research background in biology. Therefore, the following comments mainly evaluate the paper from the perspectives of its motivation, algorithmic aspects, and experimental methodology.
-
From a purely algorithmic perspective, some of the conclusions presented in this paper have already been discussed in many previous works and are fairly common observations[1][2][3]. For example, Section 4.2 argues that acquiring knowledge does not necessarily mean that the knowledge can be effectively utilized in analytical tasks. Figure 2 also shows that models after SFT, SD, and OPD perform worse than the base model on analytical tasks. In fact, this observation is quite common. After the above training procedures, models are prone to catastrophic forgetting, resulting in degraded performance on the original tasks compared with the base model. In addition, it can be observed that GRPO effectively mitigates forgetting and maintains a similar capability to the base model, potentially due to its KL-divergence-based alignment. Similarly, in Section 6.1, the authors show that knowledge replay can prevent capability forgetting, which is also a commonly observed conclusion rather than a new insight.
-
In addition, Section 4.2 presents the core conclusion that ``acquiring knowledge does not mean that the knowledge can be effectively utilized in analytical tasks.'' However, the paper does not provide a detailed analysis of why this happens. Why can the acquired knowledge not be effectively utilized? What causes this phenomenon? For example, is it caused by catastrophic forgetting after SFT? The authors mainly restate the experimental observations without providing the underlying reason that readers are most concerned about.
-
In Section 5.1, the authors state that knowledge portability depends on the required operation, but they do not clearly explain what these operations refer to. For example, do these operations refer to algorithm-level techniques for preventing knowledge forgetting, such as knowledge replay, gradient clipping, or regularization? The authors need to further clarify what exactly is meant by these operations.
-
In addition, from an experimental perspective, the authors only conduct experiments on 9B-scale models, lacking an analysis of knowledge transfer across models of different scales. The paper also lacks detailed demonstrations of the training process. For example, during GRPO training, how is the reward function designed for the biological domain? For SFT, how does the loss curve change with training steps? Similarly, for OPD, what are the parameter sizes of the teacher and student models? Is the teacher model trained with GRPO, or is the base model directly used as the teacher? These detailed experimental settings are critical for reproducing the core conclusions of the paper.
[1] Luo Y, Yang Z, Meng F, et al. An empirical study of catastrophic forgetting in large language models during continual fine-tuning[J]. IEEE Transactions on Audio, Speech and Language Processing, 2025.
[2] Li H, Ding L, Fang M, et al. Revisiting catastrophic forgetting in large language model tuning[C]//Findings of the association for computational linguistics: EMNLP 2024. 2024: 4297-4308.
[3] Abdi I, Gupta A, Mok M, et al. Evolutionary Strategies lead to Catastrophic Forgetting in LLMs[J]. arXiv preprint arXiv:2601.20861, 2026.
Questions for authors
See above weakness.
2 comments
Dear Reviewer opnS,
The discussion period ends in a few hours, and we have not received a response to our rebuttal. Your main concerns are addressed by new results in the revised manuscript:
- Catastrophic forgetting (W1): forgetting explains the drop below the Base model, not the missing gain. Once replay removes it, trained facts are recalled 35–38 points more accurately than held-out ones, yet the analyses that depend on them are no more accurate (−1.4 pp, pooled over two seeds).
- Mechanism (W2, W3): with the analysis trajectory held identical up to the final decision, training on the fact shifts the probability of fact-consistent decisions by only +0.1 points; splitting IE-SP into two lookups raises accuracy from 0.690 to 1.000.
- Model scale (W4): Qwen3.5-4B reproduces the main results of Sections 5.1 and 5.2.
Engaging with the authors' response is part of the review process, and we would appreciate the same level of care that went into our revision!
As the discussion period is coming to a close, we would be grateful to know whether our response and the additional experiments have addressed your concerns, and we would be happy to clarify any remaining questions.