- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes PRINCIPIA, an LLM-based framework for constructing and refining scientific knowledge during test-time discovery. It generates causal hypotheses by comparing related inputs and their observed outcomes, then generalizes recurring patterns into principles through clustering. These hypotheses and principles guide subsequent optimization or prediction, and are updated based on new observations. Experiments on molecular optimization, protein engineering, gene perturbation, and molecular property prediction show improvements over several LLM-based baselines.
Critical strengths
- The framework provides a structured way to organize and update knowledge from experimental observations, without requiring additional model training.
- The experiments cover a diverse range of scientific tasks, with promising improvements over relevant baselines such as MolLEO, ExLLM, and DeepEvolve.
- The paper includes useful ablation studies on comparative hypotheses, principle construction, revision, and retrieval, along with detailed implementation descriptions and prompts.
Critical weaknesses
- The generated hypotheses and principles are not systematically validated for scientific correctness. The qualitative examples are encouraging, but they provide limited evidence that the inferred relationships are genuinely causal or scientifically reliable.
- Although the paper includes an ablation without principle construction, it does not compare against directly retrieving historical input-output comparisons without LLM-generated hypotheses in the optimization tasks. This would help clarify whether the hypothesis generation step provides additional benefits beyond access to relevant past observations.
- The methodological contribution appears somewhat incremental. Prior work such as LLM4SD and DESRO has explored deriving scientific rules from molecular observations, while PiFlow and PiEvo have investigated principle-guided scientific discovery and iterative refinement. PRINCIPIA organizes these ideas into a structured test-time memory framework, but its additional technical contributions appear relatively limited.
Questions for authors
- Appendix C.1 uses PiFlow's 24-iteration protocol, while other methods are evaluated with fixed oracle budgets. Could the authors clarify how many oracle evaluations PiFlow actually performs and how Avg@100 is computed in this setting?
- The efficiency analysis focuses on oracle evaluations, but PRINCIPIA introduces additional LLM calls for hypothesis generation, principle construction, retrieval, and revision. Could the authors report the total token usage or runtime compared with the baselines, to clarify whether the reduced oracle budget translates into lower overall computational cost?
6 comments
Initial Response by Authors (2/3)
Comment 3: The methodological contribution appears somewhat incremental. Prior work such as LLM4SD and DESRO has explored deriving scientific rules from molecular observations, while PiFlow and PiEvo have investigated principle-guided scientific discovery and iterative refinement. PRINCIPIA organizes these ideas into a structured test-time memory framework, but its additional technical contributions appear relatively limited.
Response 3: Our main contribution is test-time hypotheses-to-principle construction, which couples principle formation with the continual revision of its underlying explanatory hypotheses. We highlight two distinctions from prior work:
-
Revision under incomplete evidence. LLM4SD derives rules for predictive features, while DESRO derives reasoning supervision for training. During discovery, limited observations can yield explanations that overlook relevant factors or overgeneralize beyond their supporting evidence. PRINCIPIA retains these explanations as revisable knowledge units. When new observations become available, it revisits the hypotheses used in subsequent decisions, updates their explanations while retaining the original comparative evidence, and reconstructs principles from the updated hypotheses.
-
Explanatory hypotheses as the basis of principles. In PiFlow and PiEvo, hypotheses primarily refer to experimental candidates proposed under guiding principles. PiEvo also evolves these principles using experimental feedback. PRINCIPIA maintains comparative explanatory hypotheses about the relationships between input differences and observed outcomes. These explanations remain individually usable and revisable after their recurring relationships are generalized into principles. Revising them therefore updates the explanatory basis from which principles are reconstructed.
Therefore, our contribution is this coupled process of comparative explanation, generalization, and evidence-driven revision throughout scientific discovery.
Comment 4: Appendix C.1 uses PiFlow's 24-iteration protocol, while other methods are evaluated with fixed oracle budgets. Could the authors clarify how many oracle evaluations PiFlow actually performs and how Avg@100 is computed in this setting?
Response 4: We would like to clarify that PiFlow’s generation procedure is different from MolLEO-based models: it generates and evaluates candidates sequentially, while MolLEO-based models leverage population-based genetic algorithms. We adopted the original PiFlow setting of 24 iterations for molecular optimization and protein engineering. The average numbers of unique oracle evaluations were 15.73 and 20.67, respectively. Therefore, Avg@100 and Avg@50 average all available candidates. For gene perturbation, Hit Ratio uses the same number of distinct genes across methods.
We further examined PiFlow under the same maximum oracle-evaluation budgets as PRINCIPIA, conducting three runs per task. Because PiFlow frequently generated duplicate sequences in the GB1 task of protein engineering, we supplemented its prompts with previously proposed sequences and instructions to avoid proposing them again. This provides a potentially unfair advantage to PiFlow, since PRINCIPIA was not given the same explicit exclusion list. Even with this additional support, as shown in Table C.3 and C.4, PRINCIPIA achieves higher performance on average, indicating that PRINCIPIA still outperforms PiFlow under the same oracle-evaluation budgets. We have updated Tables 1 and 2 in the revised paper to report these results under matched maximum oracle-evaluation budgets.
| Method | JNK3 | GSK3B | Median1 | Median2 | Valsartan Smarts | Avg. | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Avg@100 | AUC@10 | Avg@100 | AUC@10 | Avg@100 | AUC@10 | Avg@100 | AUC@10 | Avg@100 | AUC@10 | AUC@10 | |
| PiFlow | 0.158 | 0.254 | 0.184 | 0.275 | 0.251 | 0.283 | 0.157 | 0.164 | 0.000 | 0.000 | 0.195 |
| Ours | 0.448 | 0.350 | 0.617 | 0.452 | 0.298 | 0.284 | 0.220 | 0.203 | 0.117 | 0.106 | 0.279 |
Table C.3: Molecule optimization under matched maximum oracle budgets for three runs.
| Method | Syn-3bfo | AAV | GB1 | Scharenberg22 | IFNG | Avg. | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Avg@50 | AUC@10 | Avg@50 | AUC@10 | Avg@50 | AUC@10 | Hit Ratio | AUC@10 | Hit Ratio | AUC@10 | AUC@10 | |
| PiFlow | -29.813 | 0.164 | 0.140 | 0.101 | 0.468 | 0.462 | 0.289 | 0.557 | 0.111 | 0.682 | 0.393 |
| Ours | 0.084 | 0.742 | 0.681 | 0.527 | 0.398 | 0.383 | 0.308 | 0.630 | 0.114 | 0.742 | 0.605 |
Table C.4: Protein engineering results under matched maximum oracle budgets. Gene perturbation results are adapted from the original evaluation, which already used matched maximum oracle budgets.
Due to the character limit, the response continues below.
Initial Response by Authors (3/3)
Comment 5: The efficiency analysis focuses on oracle evaluations, but PRINCIPIA introduces additional LLM calls for hypothesis generation, principle construction, retrieval, and revision. Could the authors report the total token usage or runtime compared with the baselines, to clarify whether the reduced oracle budget translates into lower overall computational cost?
Response 5: Our primary focus on oracle efficiency is mainly due to real-world scientific discovery settings where wet-lab experiments are substantially more expensive and time-consuming than LLM inference. In other words, while PRINCIPIA leverages additional LLM computation in constructing and updating scientific knowledge, it enables to reduce the number of experimental evaluations required for discovery.
To quantify this trade-off, we report total LLM calls, token usage, API cost, and LLM wall-clock time at matched optimization performance on Syn-3bfo, using the performance targets in Fig. 5. We additionally estimate wet-lab overhead from the oracle counts, assuming $50 and two days per evaluation with sequential execution [1,2,3]. As shown in Table C.5, although PRINCIPIA incurs higher LLM costs, this additional expense is small compared with the estimated savings from fewer experimental evaluations. Consequently, investing in knowledge construction reduces estimated total cost and time, supporting the practical efficiency of PRINCIPIA in settings where experimental evaluations dominate the discovery budget.
| Method | Token Usage (million) | LLM Calls | LLM-API Cost ($) | Wet-lab + LLM-API Cost ($) | LLM Wall-Clock (day) | Wet-lab Runtime + LLM Wall-Clock (day) |
|---|---|---|---|---|---|---|
| MolLEO | 4.0 | 848 | 7.63 | 50,007.63 | 0.033 | 2,000.033 |
| Ours | 4.6 | 657 | 3.33 | 7,003.33 | 0.010 | 280.010 |
| ExLLM | 4.0 | 873 | 7.33 | 50,007.33 | 0.035 | 2,000.035 |
| Ours | 27.9 | 2,804 | 16.48 | 22,316.48 | 0.040 | 892.040 |
| WebSearch | 5.5 | 1760 | 9.14 | 50,009.14 | 0.057 | 2,000.057 |
| Ours | 6.2 | 825 | 4.29 | 8,204.29 | 0.012 | 328.012 |
| DeepEvolve | 9.1 | 5,040 | 12.78 | 50,012.78 | 0.154 | 2,000.154 |
| Ours | 4.6 | 657 | 3.33 | 6,903.33 | 0.010 | 276.010 |
Table C.5: Cost comparison at matched performance.
[1] Kosuri, Sriram, and George M. Church. "Large-scale de novo DNA synthesis: technologies and applications." Nature methods 11.5 (2014): 499-507.
[2] Wittmann, B. J., et al. "evSeq: Cost-effective amplicon sequencing of every variant in a protein library. ACS Synth Biol 11: 1313–1324." 2022,
[3] Fowler, Douglas M., Jason J. Stephany, and Stanley Fields. "Measuring the activity of protein variants on a large scale using deep mutational scanning." Nature protocols 9.9 (2014): 2267-2284.
We thank the reviewer for your time and feedback and hope that our responses have addressed your concerns. If you have any further questions, we are happy to address them. If your concerns are appropriately addressed, we hope you kindly consider updating the rating accordingly.
Initial Response by Authors (1/3)
Thank you for your constructive comments. We appreciate your recognition of the following strengths:
We address your concerns below.
Comment 1: The generated hypotheses and principles are not systematically validated for scientific correctness. The qualitative examples are encouraging, but they provide limited evidence that the inferred relationships are genuinely causal or scientifically reliable.
Response 1: Unfortunately, scientific problems are still open ended and causal relationships underlying scientific problems may not be fully established, limiting to systematically validating scientific correctness of generated hypotheses and principles. Literature search can identify supporting or conflicting evidence where relevant studies exist. However, due to a lack of literature, their validation could not be rigorous for specific inputs or settings. Instead, we adopt qualitative analysis to see the plausibility of generated hypotheses and principles and show the procedure of PRINCIPIA including the revision.
To mitigate the limitations of literature-based validation, we conduct additional experiments to assess the scientific correctness of the generated principles. Specifically, we evaluate whether principles constructed in one discovery run improve predictions of fitness-change directions on held-out comparisons from separate runs. As shown in Table C.1, providing these principles consistently improves prediction, providing quantitative evidence supporting the correctness of the input-outcome relationships captured by the generated principles.
Table C.1: Fitness direction prediction accuracy (%) with and without PRINCIPIA principles
On the other hand, in real-world drug discovery, limited observations can leave hypotheses and principles incomplete, even when developed by experts. To address this practical constraint, we propose a revision algorithm that reassesses previously used hypotheses as new observations become available. Therefore, our contribution within Applications to Sciences focuses on an algorithmic framework for continually constructing and updating scientific knowledge under real-world scenarios where observations do not fully cover the input space and empirical hypotheses and principles are incomplete.
Comment 2: Although the paper includes an ablation without principle construction, it does not compare against directly retrieving historical input-output comparisons without LLM-generated hypotheses in the optimization tasks. This would help clarify whether the hypothesis generation step provides additional benefits beyond access to relevant past observations.
Response 2: We appreciate your insightful suggestion. We further compare our comparative causal hypotheses with historical input-output pairs. As shown in Table C.2, using comparative causal hypotheses outperforms using historical comparisons, suggesting that explaining which input characteristics may contribute to observed outcomes provides useful guidance beyond access to past observations alone.
Table C.2: Top-10 AUC comparison between using simple input-outcome pairs and causal hypotheses. We report the mean over three independent trials.
Due to the character limit, the response continues below.