- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EvidenceDeliberationBench-Biology (EDB-Bio), a benchmark for evaluating whether language models can assess and update beliefs about biological hypotheses given sequential experimental evidence. The benchmark covers drug-perturbation transcriptomics, CRISPR dependency screens, and GWAS, with evidence pools ordered by biological relevance. Models output probabilities after progressively receiving evidence, enabling evaluation of Brier score, calibration, discrimination, and evidence-weight updates. The authors find that frontier and open-weight models can often rank supported hypotheses above unsupported ones but are poorly calibrated, with performance close to a base-rate forecaster under Brier score. They post-train Qwen3-30B-A3B variants using proper scoring rules, obtaining Ibis0 models with improved calibration and discrimination. They further analyze ordering effects and overreaction to evidence, and use the calibrated model in an expected-information-gain experiment-selection setting, where the proposed approach obtains substantially more information per experiment than prompted baselines.
Critical strengths
EDB-Bio provides a well-motivated task formulation for evaluating probabilistic scientific reasoning rather than only categorical answer accuracy. The use of proper scoring rules, calibration metrics, AUROC, and weight-of-evidence analyses gives a more complete view of model behavior.
The paper combines benchmark evaluation with a downstream sequential experimental-design task. The analysis of evidence ordering, overreaction, and information gain provides useful insight into how models revise beliefs as evidence accumulates.
Critical weaknesses
The data split does not fully guarantee target-level independence. The authors report that held-out test measurements appear as evidence stubs in training or validation pools for 0.6% of Tahoe, 10.1% of DepMap, and 53.2% of GWAS test episodes. Although the authors report relatively small aggregate changes after removing overlapping episodes, the high GWAS overlap makes it difficult to rule out target-, entity-, or association-level memorization and weakens the interpretation of the benchmark as testing generalization to unseen evidence.
The benchmark is a highly structured and deliberately balanced approximation to scientific evidence. Evidence stubs are summaries, labels are binary support criteria, and every domain has a 0.5 class prior. It therefore remains unclear how well the observed calibration and experiment-selection gains transfer to free-text literature, naturally imbalanced hypotheses, conflicting studies, variable experimental costs, or real scientific decision-making.
Questions for authors
Could the authors train and evaluate the main models on a strictly leakage-controlled split, constructed by excluding all measurements used as evidence stubs across splits? In particular, a GWAS evaluation with no target measurement overlap would substantially clarify the generalization claim.
Could the authors report performance stratified by overlap status and by the degree of entity or association similarity between test targets and training evidence? This would help determine whether the reported gains depend on memorization or near-duplicate evidence.
How sensitive are the calibration and EIG results to the artificially balanced 0.5 prior? Please discuss or evaluate settings with naturally varying base rates, since calibration under a fixed balanced prior may not transfer directly to scientific forecasting.
Please clarify how the evidence-stub construction and expert relevance hierarchy would extend to free-text studies with heterogeneous reporting quality, contradictory results, missing methodological details, and study-level biases.
The related-work discussion would benefit from positioning EDB-Bio alongside recent work on biomedical post-training, cellular-perturbation reasoning, and sequential evidence selection, including: “Aligning LLMs with Biomedical Knowledge using Balanced Fine-Tuning,” “PertMind: Eliciting Emergent Biological Reasoning in LLM via Reinforcement Learning on Cellular Perturbation Data,” “Contract Memory Compiler: Resolve, Then Traverse.” The first is relevant to uncertainty-aware biomedical post-training, the second to perturbation-based biological reasoning, and the third to state-dependent evidence retrieval for sequential deliberation.
The EIG comparison involves both the belief model and a particular experiment-selection estimator. Can the authors separate the contribution of calibration training from the contribution of the EIG estimator and simulated-outcome model through additional matched ablations?
2 comments
Dear Reviewer pd31,
Thank you again for your thoughtful review. With the discussion period closing today, we wanted to follow up on the concerns you raised. In response to your comments, we ran new EIG ablations and also retrained our model on a reconstructed version of the dataset with no target leakage. We also wrote up a clarification of what we consider novel beyond the benchmark itself.
Could you let us know whether this addresses your concerns? Even a brief reply before discussion closes would be appreciated.
Best, The EDB-Bio authors
We thank the reviewer for their thoughtful review and suggestions. Below, we respond to issues raised.
“The data split does not fully guarantee target-level independence.“ & question “Could the authors train and evaluate the main models on a strictly leakage-controlled split”
We agree that the original data split does not guarantee target-level independence. Based on this feedback, we retrained Ibis0 (classifier) as a representative model, and also evaluated two Qwen3 baselines (non-thinking and thinking) to check whether the new split is easier than the original. On the restratified dataset, the Ibis0 classifier's macro Brier score and AUROC fall within the 95% bootstrap intervals of the original. Building on the pre-existing A.7 analysis, this exercise demonstrates that (1) the technical concern around overlap in EDB-Bio has now been fully resolved, and (2) the impact of this issue on the original results was within measurement error. Further details are included in a dedicated response above (link).
“The EIG comparison involves both the belief model and a particular experiment-selection estimator. Can the authors separate the contribution of calibration training from the contribution of the EIG estimator and simulated-outcome model through additional matched ablations?”
We agree with the reviewer that our primary BED analysis would be strengthened with further ablations to isolate the effects of the EIG procedure and the improved underlying belief model. Based on this feedback, we ran additional experiments using the same EIG procedure as in our main Ibis0-EIG-simulate results but with frontier models as the underlying belief model during EIG estimation. We found that frontier models used in this way produce largely unhelpful EIG estimates, resulting in BED performance that only marginally improves over the random policy. These results directly address question 3, clarifying that the value in Ibis0-EIG is in both the underlying belief model’s superiro brier score and the dedicated EIG calculus at inference time. We have revised the manuscript to include these new results. Further details are included in a dedicated response above (link).
“Could the authors report performance stratified by overlap status”
Fig. 6 (Appendix 7) reports performance stratified by overlap status. As shown by the “filtered” bars, the effect of the overlap is minimal.