Beyond Hypothesis Recovery: Evaluating Scientific Hypothesis Generation through Evidence-Hypothesis Transitions
Abstract
Scientific hypothesis generation is commonly evaluated by whether a model recovers a correct or novel historical hypothesis from supplied evidence. For modern foundation models, endpoint recovery is non-identifying: the same hypothesis may arise from evidence-conditioned reasoning or parametric access to the historical relation. Temporal cutoffs and evidence interventions reduce this ambiguity but retain a static evidence-to-hypothesis endpoint as the evaluation target. We introduce HypoTrans, a benchmark of evidence-conditioned scientific-state transitions reconstructed from cell-biological literature. Given a prior state and newly documented formation evidence, with the later target state withheld, a model generates a structured hypothesis state. Typed evidence–hypothesis provenance separates formation from validation evidence, and four continuity gates support documentary validation. We characterize changes using emergence, directional alignment, revision where applicable, and evidence attribution; matched-irrelevant-evidence selectivity is not identified by the paired conditions used here. In an earlier collection, closed-book access and endpoint recovery were positively associated (, , state–model pairs); access and update-content scores were also positively associated (, , transition–model pairs). These are separate descriptive analyses on different samples, not a test that one task is superior. In a new evaluation of five models, strict exact emergence was zero, lexical mechanism-level overlap ranged from 0.050 to 0.092, and categorical directional alignment from 0.041 to 0.081; semantic correctness was not adjudicated. The benchmark provides a behavioral measure of correspondence between model-state changes and documented transitions, but does not establish internal evidence use or selective updating.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.