acceptodds
Under review as a conference paper at ICLR 2027

IdeaPrism: Beyond Ground-Truth Matching for Open-Ended Hypothesis Evaluation

Abstract

AI scientists generate open-ended hypothesis sets in which several scientifically plausible explanations may fit the same evidence, including mechanisms absent from a predefined target hypothesis. Many existing evaluations, however, score outputs by matching them to designated discoveries or by asking human or LLM judges for direct quality assessments. Direct ratings introduce evaluator dependence, while target matching can miss alternative hypotheses and obscure the breadth of a generated set. We introduce IDEAPRISM, a framework for open-ended evaluation of scientific hypothesis generation. IDEAPRISM characterizes hypothesis sets along three dimensions. Exploration Novelty measures how rarely a mechanism appears in a fixed reference population, with scores adjusted for estimated reference coverage. Diversity measures the effective number of scientific directions represented in a hypothesis set. Value measures the change in a fixed reasoner's accuracy on hidden-evidence questions when conditioned on a hypothesis. We evaluate IDEAPRISM on 61 topics spanning 13 scientific subfields. Controlled experiments assess reference-coverage estimation, effective-direction recovery, and hypothesis-conditioned prediction. We then apply the framework to ten AI-scientist scaffolds. The resulting profiles show that improvements in reference-relative exploration or predictive utility do not necessarily coincide with broader exploration across scientific directions. IDEAPRISM complements target-recovery benchmarks by providing an operational, multidimensional characterization of generated hypothesis spaces.

Primary area:
datasets and benchmarks
Keywords:
AI scientists, scientific hypothesis generation, AI for science, evaluation benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6zUBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces IdeaPrism, a framework for evaluating open-ended scientific hypothesis sets through three complementary measurements. Exploration Novelty measures mechanism rarity relative to a fixed population of five reference generators, scaled by estimated reference coverage. Diversity uses structured scientific attributes and spectral entropy to estimate effective direction counts for mechanisms, interventions, predictions, and experiments. Value measures the change in a fixed reasoner's accuracy on hidden-evidence questions when conditioned on a hypothesis.

Critical strengths
  1. This paper addresses a significant limitation in current benchmarks for scientific hypotheses: the exclusive focus on whether established scientific findings can be "reproduced," which fails to capture the breadth or practical utility of alternative hypotheses. By separately evaluating "novelty relative to a reference baseline," "diversity across constituent dimensions," and "predictive utility," the researchers can more clearly examine the trade-offs between these metrics—trade-offs that are often obscured when reduced to a single, aggregate score. Furthermore, the paper provides precise definitions for "novelty" and "value," carefully avoiding any implication that these terms are synonymous with absolute scientific originality or impact.

  2. The metrics proposed in this work are grounded in clear operational definitions and supported by targeted validation methods. The "diversity kernel" is constructed to ensure positive semi-definiteness, with its limiting cases offering meaningful interpretations; meanwhile, the "value" metric is assessed by comparing predictive outcomes with and against the inclusion of a specific hypothesis. Through coverage calibration studies, controlled directional replication experiments, and the evaluation of treatment effects across different "reasoners," the paper provides a robust empirical foundation for the performance of these metrics. Compared to approaches that rely solely on unconstrained quality scores, this framework offers superior interpretability and auditability.

Critical weaknesses
  1. The existing validation does not establish the reliability of this framework when it comes to evaluating "naturally generated and scientifically plausible" alternatives. Experiment E1 only validates the estimation of "unseen mass" within a selected reference population; it does not validate the effectiveness of the resulting "novelty" scores for evaluating alternative mechanisms. Experiment E2 assesses a collection of restated, previously published ideas. In Experiment E3, research hypotheses are constructed based on latent research findings; for those hypotheses deemed "supported," the authors explicitly verify whether they indeed lead to the predetermined correct answers (see Appendices D and F). While the resulting rankings serve as a useful sanity check, they do not provide sufficient evidence that the framework can effectively distinguish between competing scientific hypotheses—particularly those that are incomplete or indirect. Furthermore, although the study spans multiple disciplines, the human evaluation (concerning "novelty" and "diversity" metrics) is limited to AI-generated content. While I do not require these operational metrics to measure absolute scientific quality, substantiating the paper’s core contributions demands stronger evidence linking these metrics to their intended application—the evaluation of generated hypothesis sets.

  2. Regarding the system comparisons, the paper lacks sufficient information on effective sample sizes and uncertainty. Section 6 and Tables 8–11 provide only point estimates for each system. However, the "diversity" metric employs filtering criteria specific to certain components, and the "value" metric applies a "headroom screen" tailored to specific reasoning models; moreover, cases with incomplete or ambiguous outputs were excluded from scoring (see Appendix C.5). The paper does not explicitly list the topics retained for each system and metric, nor does it clarify whether the comparisons are based on an identical set of topics. Appendix D mentions 17 overlapping topics between two groups of reasoning models, yet it fails to specify the total size or composition of each group. In the absence of these denominators, a summary of missing data, and estimates of pairwise uncertainty, it is difficult to disentangle the effects attributable to "workflow" from those resulting from differences in topic composition or sampling variation. This directly impacts the strength of the evidence supporting the "scaffold profiles" and the ranking results presented in the paper.

Questions for authors
  1. How many topics were covered when assessing each system across the dimensions of "Novelty," "Diversity," and "Value"? Please specify the "headroom cohorts" defined for each specific reasoner, the data excluded due to incomplete runs or invalid responses, and whether the same intersection of topics was used when comparing "scaffolded" reasoning against "direct" reasoning. If the evaluation cohorts differ, do the primary conclusions still hold when restricted to a matched set of topics?

  2. Could you provide paired topic-bootstrap confidence intervals for the differences between "scaffolded" and "direct" reasoning, and quantify the stability of the reported rankings? Additionally, please clarify the number of independent generation runs performed by each system for each topic, and specify which sources of variation are accounted for in your uncertainty estimates.

  3. Beyond the manually defined A–D treatment conditions, are there independently evaluated samples of system outputs that demonstrate the reasonableness of these metrics—for instance, in cases involving plausible alternative mechanisms, scientifically incorrect mechanisms that rarely appear in reference materials, or hypotheses with predictive scopes extending beyond the specific "hidden-question set"? Providing cross-disciplinary examples along with expert evaluations would significantly strengthen my confidence in the intended use of this benchmark.

4 comments

Response to Question 1: Effective Sample Sizes, Headroom Cohorts, Missing Data, and Matched-Topic Robustness

We thank the reviewer for raising this important concern. We agree that reporting effective sample sizes and evaluation cohorts explicitly is essential for interpreting system-level comparisons. We would like to clarify that all scaffolded and Direct LLM configurations were evaluated on identical final topic sets within each metric, rather than independently selected topic subsets.

(1) Effective sample sizes across evaluation dimensions.

All 22 configurations (11 workflows × 2 generator backbones) were evaluated on the same 51 held-out topics for Novelty and all four Diversity components (Mechanism, Intervention, Prediction, and Experiment).

For Diversity, hypotheses with unspecified required scientific attributes are excluded only from the corresponding component calculation. Importantly, this component-level filtering did not result in any topic-level exclusions: all 51 topics remained scorable for every workflow and Diversity component.

(2) Reasoner-specific headroom cohorts.

Value applies the reasoner-specific headroom criterion S−h≤0.85S^{-h}\leq 0.85. Among the 51 held-out topics, Gemini-2.5-Flash retains 42 topics, while GPT-4.1 retains 21 topics, with 17 topics shared by both cohorts, consistent with the common-cohort analysis already reported in Appendix D.

Crucially, all Section 6 system comparisons use Gemini-2.5-Flash as the fixed Value reasoner. Thus, all 22 configurations, including Direct LLM, are evaluated on the same 42 Gemini-eligible topics. GPT-4.1 is used separately for E3 validation and does not affect the system-comparison cohort.

Table 1. Effective Evaluation Cohorts Across Dimensions

EvaluationInitial TopicsRetained TopicsShared Across Systems?
Novelty5151Yes
Diversity (M/I/P/E)5151Yes
Value (Gemini; Section 6)5142Yes
Value (GPT-4.1; E3)5121Separate validation cohort
Gemini ∩ GPT-4.15117Cross-reasoner sensitivity

(3) Incomplete runs and invalid responses.

Appendix C.5 already documents our treatment of incomplete runs and ambiguous responses. We further clarify that no additional topic-level exclusions occurred due to incomplete generation runs or invalid Value responses. Consequently, the final effective denominators remain identical across workflows: 51 topics for Novelty and each Diversity component, and 42 topics for Value.

While incomplete scientific specifications affect the number of hypotheses contributing to individual Diversity components, they do not change the topic cohorts used for system comparisons. We will make the distinction between hypothesis-level filtering and topic-level eligibility more explicit and provide corresponding specification-coverage statistics and topic identifiers.

(4) Matched-topic comparisons and robustness of the main conclusions.

We emphasize that the evaluation cohorts differ across metrics, but not across systems within the same metric. All scaffolded and Direct LLM configurations use exactly the same 51 topics for Novelty and Diversity and the same 42 Gemini-eligible topics for Value.

Therefore, the original scaffold–Direct comparisons are already matched-topic comparisons. Restricting each comparison to the intersection of valid topics does not alter the evaluated topic sets or the reported numerical results.

Importantly, the main conclusions remain unchanged under matched-topic evaluation. Scaffolds exhibit positive aggregate improvements in Novelty (Qwen: +0.079; GLM: +0.156) and Value (Qwen: +0.031; GLM: +0.040), whereas Mechanism Diversity shows backbone-dependent behavior (Qwen: +1.168; GLM: −0.067).

These findings demonstrate that the observed scaffold profiles are not artifacts of differences in topic composition. Instead, they reflect distinct patterns of exploration novelty, scientific-direction diversity, and predictive utility across AI-scientist workflows.

Overall, the concern stems from insufficiently explicit reporting of effective denominators rather than unmatched evaluation cohorts. We will revise the manuscript to clearly document the complete headroom cohorts, topic-level exclusion counts, component-level specification coverage, and matched-topic evaluation protocol.

Response to Question 2: Statistical Uncertainty, Repeated-Run Robustness, and Ranking Stability

We thank the reviewer for highlighting the importance of quantifying statistical uncertainty and assessing ranking stability. We have conducted 10,000 paired topic-level bootstrap resamples and three independent generation runs per system configuration. The detailed statistical results are also reported in our response to Reviewer CBef's Question 2.

(1) Bootstrap confidence intervals. We performed paired bootstrap resampling over identical evaluation cohorts across systems: 51 held-out topics for Novelty and Diversity, and 42 Gemini-eligible topics for Value.

The scaffold–Direct improvements remain positive for Novelty (Qwen: +0.079, 95% CI [0.043, 0.115]; GLM: +0.156, [0.104, 0.208]) and Value (Qwen: +0.031, [0.012, 0.050]; GLM: +0.040, [0.018, 0.062]).

For Mechanism Diversity, the results are backbone-dependent: Qwen shows a positive improvement (+1.168, [0.921, 1.423]), whereas GLM's confidence interval includes zero (−0.067, [−0.195, 0.061]). These findings support the positive aggregate Novelty and Value results while preserving the observed differences in Diversity.

(2) Repeated-run robustness. Across three independent generation runs, the mean scaffold–Direct improvements remain consistent with the original results:

  • Novelty: Qwen +0.076 ± 0.012; GLM +0.151 ± 0.019.
  • Value: Qwen +0.030 ± 0.008; GLM +0.038 ± 0.010.
  • Mechanism Diversity: Qwen +1.253; GLM −0.005.

Thus, the positive Novelty and Value trends persist across repeated generations, while Mechanism Diversity remains substantially stronger for Qwen than for GLM.

(3) Ranking stability. We additionally compared the rankings of all 11 workflows within each generator backbone across the three runs. Across Novelty, Mechanism Diversity, and Value, the average pairwise Spearman correlation is 0.94, Kendall correlation is 0.85, and top-three overlap is 88.9%. The corresponding ranges are 0.91–0.96, 0.80–0.89, and 77.8%–100%, respectively.

These results indicate that the overall workflow rankings are highly consistent across independent runs, although individual pairwise ordering differences should not be interpreted as statistically significant without further testing.

Overall, the paired bootstrap, repeated-generation experiments, and ranking analysis provide complementary evidence for the robustness of our main findings. In particular, the positive aggregate Novelty and Value improvements are stable, whereas the Diversity results remain backbone-dependent. We will incorporate these uncertainty estimates and reproducibility analyses into the revised manuscript.

Response to Question 3: Independent Expert Validation and Challenging Scientific Hypotheses

We thank the reviewer for emphasizing the importance of evaluating naturally generated scientific hypotheses beyond the controlled A–D treatments. To address this concern, we provide independent domain-expert evaluations across AI/ML and Life Sciences, together with two complementary real-generation case studies involving plausible alternative mechanisms and scientifically questionable but novel hypotheses.

(1) Independent expert validation. As reported in our additional experiments, we evaluated 100 naturally generated hypotheses across AI/ML and Life Sciences, with two independent domain experts per field.

Table 2. Independent Expert Validation of Predictive Value

DomainHypothesesExpert Agreement ↑Value–Expert Spearman ρ ↑
AI/ML500.820.71
Life Sciences500.780.65
Overall1000.800.68

The positive correlations across both domains provide external evidence that IdeaPrism's Value metric aligns with independent expert assessments of naturally generated hypotheses, rather than merely distinguishing manually constructed treatment conditions.

To further validate all three dimensions, we conducted an expert evaluation on 240 naturally generated hypotheses from four AI-scientist systems across six topics in AI and Life Sciences. Each topic was independently assessed by two corresponding domain experts under blinded conditions.

Table 3. Agreement Between IdeaPrism and Independent Domain Experts

DimensionMetricHuman–HumanIdeaPrism–Human
NoveltyPairwise Ranking Agreement ↑86.2%80.4%
DiversityEffective Direction Count MAE ↓0.310.47
ValuePredictive Informativeness Agreement ↑83.5%76.8%

These results demonstrate substantial alignment with independent expert judgments across Novelty, Diversity, and Value, complementing our earlier Value-focused validation and providing additional evidence for IdeaPrism's applicability to actual AI-scientist outputs.

(2) Case Study A: A Plausible Alternative Mechanism. We examined a hypothesis naturally generated by Virtual Scientist (Qwen2.5-72B) for the Lost in the Middle problem.

While the designated explanation emphasizes positional attention bias, the generated hypothesis proposes insufficient utilization of document structure and topical coherence as an alternative contributor to middle-position retrieval failures.

Conditioning on this hypothesis improves prediction accuracy from 7/12 to 11/12 (Value = +0.333), and independent domain experts assessed it as scientifically useful.

This case illustrates that IdeaPrism can recognize the predictive utility of plausible alternative mechanisms without requiring agreement with a predefined target explanation.

(3) Case Study B: A Novel but Scientifically Questionable Mechanism. We examined Weighted-Residual Grokking Theory, naturally generated by MultiAgent Ideator (GLM-4-32B-0414). The hypothesis claims that sufficiently influential residual pathways are necessary for grokking.

IdeaPrism assigns high Novelty (0.933) because the proposed mechanism is unmatched in the reference pool of 150 hypotheses, but negative Value (−0.250), with prediction accuracy decreasing from 10/12 to 7/12.

Importantly, Levi et al. (ICLR 2024) demonstrated analytically and numerically that grokking can occur in simple linear models without requiring residual or skip connections, challenging the hypothesis's universal necessity claim.

This case illustrates that high reference-relative novelty does not imply scientific correctness or predictive utility. IdeaPrism distinguishes these properties rather than conflating them into a single scientific-quality score.

(4) Predictive scope and incomplete hypotheses. We acknowledge that Value measures predictive utility over the evaluated hidden questions, potentially underestimating hypotheses with broader predictive implications. Similarly, incomplete scientific specifications are reflected in component-level Diversity coverage. We will clarify these limitations, particularly for incomplete or indirect hypotheses.

Overall, our cross-disciplinary expert evaluations and complementary real-generation cases provide evidence beyond the controlled A–D treatments. They demonstrate that IdeaPrism can recognize scientifically useful alternatives without target matching, distinguish novelty from predictive utility, and provide interpretable multidimensional assessments of naturally generated hypotheses.

Thank you again for reviewing our work. Before the discussion period closes, we would welcome any further questions or feedback and would be happy to clarify any points that remain unclear.

Reviewer_CBefOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces IdeaPrism, a framework for evaluating open-ended scientific hypotheses beyond ground-truth matching and direct LLM judgments. It proposes three metrics: Exploration Novelty measures mechanism rarity relative to reference LLM generations; Diversity estimates effective scientific directions using structured attributes and spectral entropy; and Value measures hypothesis-conditioned prediction gains on hidden-evidence questions. The authors construct 61 scientific topic packs, conduct controlled metric validation experiments, and evaluate 10 AI scientist scaffolds across two backbones. The results suggest that improvements in Novelty and Value do not necessarily coincide with greater Diversity.

Critical strengths
  1. The paper addresses an important problem in evaluating open-ended scientific hypotheses, where predefined ground-truth matching may overlook alternative scientific explanations.
  2. The experiments cover multiple scientific domains and 10 AI scientist scaffolds. The paired with/without-hypothesis design for measuring predictive utility is also a reasonable approach.
Critical weaknesses
  1. Overclaimed open-ended evaluation

The paper claims to move beyond ground-truth matching, yet its Value metric still relies on ground truth derived from scientific papers published after the evidence cutoff. It evaluates predictions against predefined findings rather than independently assessing open-ended hypotheses. Although the target changes from a designated hypothesis to subsequent scientific findings, the fundamental dependence on ground truth remains.

  1. Limited methodological novelty and insufficient comparison with prior work

All three proposed metrics have close precedents in existing work. Novelty follows frequency-based originality scoring (MuseScorer, 2025); Diversity applies the spectral-entropy formulation of Vendi Score (Vendi Score, 2023); and Value closely resembles hypothesis-conditioned predictive utility (HypoBench, 2025). The paper does not adequately establish its technical contributions beyond these existing approaches. In particular, Vendi Score is not cited, and Table 1 omits HypoBench, overstating the framework's methodological distinctiveness.

  1. Lack of independent validation of the proposed metrics

The three metrics rely on predefined assumptions about scientific novelty, diversity, and predictive utility, but their validity is not established through independent evaluation. In particular, the paper lacks comparisons with domain-expert judgments on actual AI scientist outputs, leaving the reliability and scientific meaning of the proposed metrics unclear.

  1. Insufficient evidence for the empirical conclusions

The scaffold comparisons do not report independent repeated runs or variance across random seeds, making it unclear whether the results are robust to generation randomness. Confidence intervals or error bars for the main comparisons are also missing. Moreover, computational budgets are not equalized, and Diversity scores are affected by differences in specification coverage. These uncontrolled factors make the reported improvements and system rankings difficult to interpret reliably.

  1. Insufficient benchmark documentation and presentation quality

Several claimed benchmark details are not fully provided in the manuscript. For example, appendix A does not list all 61 topics mentioned in Sec 3.2. And several figures repeat similar results, and Figure 4 is difficult to read and compare different agents. These issues limit the benchmark's transparency and interpretability.

  • MuseScorer: Idea Originality Scoring At Scale. EMNLP 2025.
  • The Vendi Score: A Diversity Evaluation Metric for Machine Learning. TMLR 2023.
  • HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation. arXiv, 2025
Questions for authors
  1. How do the controlled validation results generalize to independently generated scientific hypotheses? Can the authors provide external expert validation and demonstrate the reliability of the metrics on actual AI scientist outputs?
  2. Do the main conclusions remain valid after accounting for statistical uncertainty and repeated runs?
3 comments

Response to Question 1: Independent Expert Validation on Real AI-Scientist Outputs

We thank the reviewer for this valuable suggestion. To assess whether our controlled validation results generalize to actual AI-scientist outputs, we conducted an independent domain-expert evaluation covering 6 topics from AI and Life Sciences, 4 AI-scientist systems, and 10 hypotheses per system per topic (240 hypotheses in total).

Each topic was independently evaluated by two domain experts under blinded conditions. Experts were provided with the research question, visible evidence, and anonymized hypotheses, without access to system identities or IdeaPrism scores. We independently evaluated all three dimensions as follows.

(1) Novelty: Pairwise Mechanism Ranking

Experts compared hypothesis pairs to determine which proposed mechanism was more novel. We measured the agreement between IdeaPrism's Novelty rankings and expert preferences.

Table 1. Expert Agreement on Novelty

EvaluationPairwise Ranking Agreement ↑
Human–Human86.2%
IdeaPrism–Human80.4%

(2) Diversity: Effective Scientific Direction Counts

Experts independently grouped hypotheses within each generated set according to their scientific directions. We computed expert-derived effective direction counts and compared them with IdeaPrism's Diversity estimates using mean absolute error (MAE).

Table 2. Expert Agreement on Diversity

EvaluationEffective Direction Count MAE ↓
Human–Human0.31
IdeaPrism–Human0.47

(3) Value: Pairwise Predictive Informativeness

Experts compared hypothesis pairs based on their usefulness for predicting observable scientific outcomes, rather than overall scientific quality. We evaluated whether IdeaPrism's Value scores yielded rankings consistent with these independent judgments.

Table 3. Expert Agreement on Value

EvaluationPairwise Predictive Informativeness Agreement ↑
Human–Human83.5%
IdeaPrism–Human76.8%

Overall, IdeaPrism achieves agreement levels approaching inter-expert consistency across all three dimensions. These results provide independent external validation on actual AI-scientist-generated hypotheses, complementing the controlled experiments (E1–E3) and supporting the scientific interpretability of our operational measurements.

Response to Question 2: Statistical Uncertainty, Repeated-Run Robustness, and Ranking Stability

We thank the reviewer for highlighting the importance of statistical uncertainty and repeated-run reliability. To directly address this concern, we conducted additional bootstrap analyses and three independent generation runs for each original experimental configuration, and further evaluated system ranking stability.

(1) Statistical uncertainty and repeated-run robustness.

We performed 10,000 paired topic-level bootstrap resamples over the 51 held-out topics, preserving the paired comparisons between each AI-scientist scaffold and its corresponding Direct LLM baseline. We report 95% confidence intervals for the aggregate scaffold improvements. In addition, we independently repeated the original generation experiments three times per configuration, using the same evaluation protocol, and measured variability across runs.

Table X. Statistical Uncertainty and Repeated-Run Robustness

MetricOriginal Δ95% Bootstrap CI3 Runs (Mean ± SD)
Novelty (Qwen)+0.079[+0.043, +0.115]+0.076 ± 0.012
Novelty (GLM)+0.156[+0.104, +0.208]+0.151 ± 0.019
Mechanism Diversity (Qwen)+1.168[+0.921, +1.423]+1.253
Mechanism Diversity (GLM)−0.067[−0.195, +0.061]−0.005
Value (Qwen)+0.031[+0.012, +0.050]+0.030 ± 0.008
Value (GLM)+0.040[+0.018, +0.062]+0.038 ± 0.010

Δ denotes the average scaffold improvement relative to the corresponding Direct LLM baseline within each generator backbone.

The results show that Novelty and Value improvements remain positive for both generator backbones, with all corresponding 95% bootstrap confidence intervals excluding zero. Their repeated-run means also closely match the original estimates, indicating limited sensitivity to generation randomness.

Importantly, the additional analysis confirms the backbone-dependent behavior of Mechanism Diversity. Qwen exhibits a substantial positive improvement (+1.168; 95% CI: [+0.921, +1.423]), consistent with its three-run mean (+1.253). In contrast, GLM shows no clear aggregate improvement (−0.067; 95% CI: [−0.195, +0.061]), with a three-run mean close to zero (−0.005). This distinction reinforces our original observation that gains in Novelty and Value do not necessarily translate into gains in every Diversity dimension.

(2) System ranking stability across independent runs.

To further investigate whether the relative performance of AI-scientist workflows depends on random generation effects, we examined system rankings across the three independent runs. For each generator backbone and evaluation metric, we ranked all 11 workflows (10 AI-scientist scaffolds and Direct LLM) using their topic-averaged scores. We then computed Spearman rank correlation, Kendall rank correlation, and Top-3 overlap for each pair of runs, averaging over the three run pairs.

Table Y. System Ranking Stability Across Three Independent Runs

MetricBackboneSpearman ρ ↑Kendall τ ↑Top-3 Overlap ↑
NoveltyQwen0.940.8588.9%
NoveltyGLM0.960.89100.0%
Mechanism DiversityQwen0.920.8288.9%
Mechanism DiversityGLM0.950.8788.9%
ValueQwen0.910.8077.8%
ValueGLM0.930.8488.9%
Average—0.940.8588.9%

The results demonstrate high ranking consistency across independent generations, with an average Spearman correlation of 0.94, Kendall correlation of 0.85, and Top-3 overlap of 88.9%. This suggests that the relative ordering of AI-scientist workflows is largely preserved across runs, rather than being driven by individual random generations.

(3) Implications for our main conclusions.

Together, these analyses provide complementary evidence for the reliability of our findings: (i) the aggregate Novelty and Value improvements remain positive under topic-level uncertainty; (ii) independent generation runs reproduce the main performance patterns, including the backbone-dependent behavior of Mechanism Diversity; and (iii) system rankings exhibit strong cross-run consistency.

Crucially, our central conclusion is not that every AI-scientist scaffold improves every metric, nor that all individual pairwise rankings are statistically distinguishable. Rather, IdeaPrism reveals systematic, multidimensional differences across AI-scientist workflows that cannot be captured by a single evaluation metric. The additional experiments strengthen this conclusion while motivating appropriate caution about small pairwise differences and unequal computational budgets.

We will incorporate these results and the corresponding statistical procedures into the revised manuscript.

We appreciate your careful review and helpful suggestions. With the discussion period nearing its end, is there anything in our responses or revised manuscript that you would like us to clarify? We would be happy to provide further details.

Reviewer_p8kxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces IdeaPrism, a framework for evaluating open-ended scientific hypothesis sets through three complementary dimensions: reference-relative exploration novelty, scientific-direction diversity, and hypothesis-conditioned predictive utility. Novelty combines mechanism frequencies in a fixed reference population with a Good–Turing coverage adjustment. Diversity applies spectral entropy to agreement kernels constructed from structured scientific attributes. Value measures the change in a fixed reasoner’s accuracy on hidden-evidence questions when given a hypothesis. The benchmark contains 61 topics across 13 scientific subfields, with system comparisons conducted on 51 held-out topics. Controlled studies evaluate the individual measurements, and experiments compare ten AI-scientist scaffolds and a direct- generation baseline under two generator backbones.

Critical strengths
  1. The evaluation framework addresses a meaningful limitation of target-recovery benchmarks. A hypothesis need not reproduce a designated discovery to offer a useful explanation. Separately reporting exploration, directional breadth, and predictive utility provides a more informative characterization of generated hypothesis sets. The distinction between reference-relative novelty and absolute scientific novelty is also appropriately acknowledged.

  2. The benchmark includes useful operational detail and controlled validation. The separation of visible and hidden evidence, the development/held-out split, and the component-level treatment of diversity make the evaluation reasonably well specified. The diversity results in Table 2 provide evidence that structured scientific attributes can improve direction recovery over several text-based alternatives, particularly for mechanisms, interventions, and predictions.

Critical weaknesses
  1. Limited methodological novelty and insufficient positioning against closely related work. Equations (6)–(7) instantiate the Vendi Score [1]: the exponential entropy of the normalized eigenvalues of a positive-semidefinite similarity matrix. Since the proposed kernel has unit diagonal, its trace normalization is equivalent to Vendi Score’s normalization. This directly relevant work is not cited or discussed; the contribution therefore lies primarily in the scientific representation and kernel construction. Likewise, LLM-based semantic grouping followed by frequency-based originality scoring is already explored in MuseScorer [2], which the paper acknowledges. The additional Good–Turing factor is constant across candidates within a topic and cannot change their within-topic ranking. E1 validates coverage estimation, but does not demonstrate that this adjustment improves novelty assessment. Evaluating hypotheses through their contribution to predictive performance also has precedents, including study in [3]. While integrating these ideas into a unified scientific evaluation protocol is useful, the paper needs clearer positioning and targeted ablations to establish the benefits beyond its existing components.

  2. Limited validation of scientific meaning beyond the operational definitions. In E3, treatment hypotheses are constructed using hidden evidence, and Appendix F explicitly describes Supported hypotheses as entailing the keyed answers and Misleading hypotheses as contradicting them. Their ordering by predictive gain is a useful sanity check, but provides limited evidence for distinguishing realistic, competing hypotheses whose scientific merit is less directly encoded in the answers. This distinction matters because hypothesis-conditioned prediction, as also studied in [3], establishes usefulness for a particular prediction task; it does not by itself establish broader scientific value. Furthermore, the human validation is narrow relative to the benchmark’s disciplinary scope: the Novelty and Diversity audits concern AI items, while question validity and Value treatments are reviewed by one annotator with a computer science Ph.D. Independent domain-expert evaluation of actual system outputs—especially cases where IdeaPrism disagrees with simpler metrics or target-recovery evaluation—would provide stronger support for the framework’s central contribution.

[1] The Vendi Score: A Diversity Evaluation Metric for Machine Learning.
[2] MuseScorer: Idea Originality Scoring At Scale.
[3] Literature Meets Data: A Synergistic Approach to Hypothesis Generation.

Questions for authors
  1. Do the authors agree that Equations (6)–(7) are an application of Vendi Score to the proposed component-agreement kernels? What contribution is intended beyond the structured representation, kernel design, and specification-coverage handling?

  2. Are there results isolating the benefit of the Good–Turing adjustment over unadjusted inverse frequency, and of the spectral score over simpler effective cluster counts using the same extracted attributes?

  3. Is there independent domain-expert validation on naturally generated hypotheses, rather than the authored A–D treatments? In particular, can the authors provide cases where IdeaPrism correctly recognizes a useful alternative to a designated target, supported by evidence beyond its own scores?

1 comment

Thank you again for your thoughtful feedback. As the discussion period draws to a close, please let us know if any questions or concerns remain; we would be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.