acceptodds
Withdrawn from ICLR 2027

Beyond Hypothesis Recovery: Evaluating Scientific Hypothesis Generation through Evidence-Hypothesis Transitions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scientific hypothesis generation is commonly evaluated by whether a model recovers a correct or novel historical hypothesis from supplied evidence. For modern foundation models, endpoint recovery is non-identifying: the same hypothesis may arise from evidence-conditioned reasoning or parametric access to the historical relation. Temporal cutoffs and evidence interventions reduce this ambiguity but retain a static evidence-to-hypothesis endpoint as the evaluation target. We introduce HypoTrans, a benchmark of evidence-conditioned scientific-state transitions reconstructed from cell-biological literature. Given a prior state and newly documented formation evidence, with the later target state withheld, a model generates a structured hypothesis state. Typed evidence–hypothesis provenance separates formation from validation evidence, and four continuity gates support documentary validation. We characterize changes using emergence, directional alignment, revision where applicable, and evidence attribution; matched-irrelevant-evidence selectivity is not identified by the paired conditions used here. In an earlier collection, closed-book access and endpoint recovery were positively associated (, , state–model pairs); access and update-content scores were also positively associated (, , transition–model pairs). These are separate descriptive analyses on different samples, not a test that one task is superior. In a new evaluation of five models, strict exact emergence was zero, lexical mechanism-level overlap ranged from 0.050 to 0.092, and categorical directional alignment from 0.041 to 0.081; semantic correctness was not adjudicated. The benchmark provides a behavioral measure of correspondence between model-state changes and documented transitions, but does not establish internal evidence use or selective updating.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
scientific discovery, hypothesis generation, biomedical benchmark, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GCcrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces HypoTrans, a benchmark for evaluating scientific hypothesis generation through evidence-conditioned hypothesis-state transitions rather than static recovery of historical hypotheses. The benchmark reconstructs scientific states from cell-biology literature, represents evidence–hypothesis provenance, and identifies historical transitions through four continuity criteria. Models are evaluated under paired evidence conditions, with performance measured through hypothesis emergence, directional alignment, revision, and evidence attribution. Experiments involving five language models report limited correspondence between generated updates and documented scientific transitions, while an earlier analysis examines the relationship between parametric accessibility and hypothesis recovery.

Critical strengths

The paper attempts to distinguish static hypothesis recovery from hypothesis-state updating and introduces a structured representation of evidence–hypothesis relationships. However, the experimental validation is insufficient to establish that the proposed benchmark measures scientific reasoning more reliably than existing approaches.

Critical weaknesses
  1. The proposed benchmark does not resolve the fundamental identifiability problem it claims to address. HypoTrans still evaluates correspondence with historical hypotheses potentially memorized by the models. Comparing outputs under two evidence conditions does not establish genuine evidence-conditioned reasoning. The authors themselves acknowledge that internal evidence use and selective updating cannot be established, substantially undermining the central contribution.

  2. The evaluation metrics are scientifically inadequate. Hypothesis emergence relies on lexical token overlap with a threshold of 0.35, while directional alignment depends on these matches. Such metrics cannot reliably assess semantic equivalence or biological correctness. Without expert-validated semantic evaluation, the reported scores are of questionable scientific validity.

  3. The benchmark ground truth lacks sufficient validation. Appendix G reports evidence–hypothesis link validity of only 0.76, while the 258 transitions are described as candidates pending expert adjudication. Human verification covers only a small subset. Heavy reliance on model-generated annotations without comprehensive expert validation seriously undermines the reliability of the benchmark.

  4. The experimental evidence fails to establish any convincing advantage over existing hypothesis-recovery benchmarks. The reported correlations (ρ = 0.373 versus 0.187) are calculated on different samples and cannot support a direct comparison. Moreover, the absence of irrelevant- and contradictory-evidence controls prevents meaningful evaluation of selective updating. The central methodological claims remain experimentally unsupported.

  5. The experimental results and benchmark construction raise substantial concerns. The abstract and Table 2 report markedly different mechanism-level and directional alignment scores without sufficient clarification, while exact emergence is zero across all models. Furthermore, 258 transitions form 255 trajectories, with only two multi-step trajectories, making the claimed evaluation of longitudinal scientific reasoning largely unsupported.

Questions for authors
  1. How does HypoTrans distinguish genuine evidence-conditioned reasoning from memorized historical transitions, given that the proposed evaluation does not establish internal evidence use?

  2. Can the authors demonstrate that the lexical matching metrics reliably capture semantic equivalence and biological correctness through expert validation?

  3. How many of the 258 transitions were independently verified by domain experts, and how do the authors address the relatively low evidence–hypothesis link validity of 0.76?

  4. Can the authors provide a controlled comparison with existing hypothesis-recovery benchmarks using identical samples and irrelevant- or contradictory-evidence interventions?

  5. How do the authors explain the discrepancies between the abstract and Table 2, and justify evaluating longitudinal scientific reasoning with almost exclusively single-transition trajectories?

Reviewer_GA2QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HypoTrans, a benchmark for evaluating how models update scientific hypotheses as new evidence arrives. It reconstructs transitions from cell-biology papers and measures hypothesis emergence, revision, directional alignment, and evidence attribution.

Critical strengths

Evaluating how models update hypotheses is a useful complement to checking whether they can reproduce known ones. Separating formation evidence from later validation is a thoughtful design choice. The explicit evidence links and continuity checks also make the benchmark easier to inspect.

Critical weaknesses

The reported results need clarification. The abstract gives directional alignment scores of 0.041–0.081, while Table 2 reports 0.273–0.625. It is unclear whether these numbers come from different datasets, scoring rules, or manuscript versions.

I am not yet convinced that the scores reliably capture scientific correctness. Mechanism matching uses a token-F1 threshold of 0.35, and the directional labels depend on these matches. Without expert assessment of model outputs, it is hard to tell whether low scores reflect scientific errors or simply different wording. Conversely, incorrect mechanisms could receive credit for using similar terminology. The construction audit does not resolve this concern.

The claim that the benchmark reduces dependence on parametric knowledge goes beyond the evidence presented. The two access correlations come from different samples. The abstract acknowledges that they do not establish superiority, but Figure 4 still concludes that the update task reduces this confound.

Questions for authors
  1. Which results should readers take as final, how many transitions are included in Table 2, and where are the stated confidence intervals reported?

  2. How well do the automatic scores agree with blinded expert judgments of generated updates, including scientifically reasonable alternatives to the historical hypothesis?

Reviewer_ktd9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces HypoTrans, a benchmark for evaluating how models change their hypothesis when new scientific evidence is provided. The paper argues that evaluating only at the endpoint of hypothesis recall is insufficient, and that evidence-conditioned transition is a better evaluation scheme. It reconstructs hypothesis states and transitions from cell-biology literature. Experiments compare five models. The results also included a small human quality assessment.

Critical strengths
  • The evaluation question is meaningful. I agree that recovering a historical hypothesis doesn't necessarily show that a model used the given evidence. Evaluating changes in hypothesis states provides a useful additional perspective on scientific reasoning, especially when models may have already been exposed to historical findings during pretraining. However, establishing proper evaluation procedure and ground truth seems very difficult here.
Critical weaknesses
  • The reliability of the benchmark construction, labels, and evaluation process is not sufficiently demonstrated. The benchmark construction relies on a massive procedure with an LLM integrated into each component. No result seems to establish correct hypothesis extraction, evidence roles, or transition labels. The human assessment also only covers a small part of the space, and the paper doesn't report the number or expertise of annotators, their disagreement, etc.
  • The overall idea of the benchmark may not adequately account for the open-ended nature of scientific discovery. The same evidence may support several reasonable hypotheses or update paths. Matching one documented historical transition therefore cannot distinguish incorrect reasoning from valid alternatives (and, if anything, falls for the same issue that the benchmark wanted to solve). It seems to me that this is a fundamental flaw of the benchmark.
Questions for authors
  1. How do you evaluate scientifically valid hypotheses or updates that differ from the historical reference?
  2. How would you guarantee the accuracy of the extracted hypothesis, evidence links, and other parts that result from an LLM component?
  3. Who performed the expert validation, and what agreement was observed among the experts?
  4. Which cases are fully verified in the final evaluation? How is the lineage counts calculated?
  5. How sensitive are the benchmark metrics to extraction errors and other variations in the benchmark process?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.