- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces IdeaPrism, a framework for evaluating open-ended scientific hypothesis sets through three complementary measurements. Exploration Novelty measures mechanism rarity relative to a fixed population of five reference generators, scaled by estimated reference coverage. Diversity uses structured scientific attributes and spectral entropy to estimate effective direction counts for mechanisms, interventions, predictions, and experiments. Value measures the change in a fixed reasoner's accuracy on hidden-evidence questions when conditioned on a hypothesis.
Critical strengths
-
This paper addresses a significant limitation in current benchmarks for scientific hypotheses: the exclusive focus on whether established scientific findings can be "reproduced," which fails to capture the breadth or practical utility of alternative hypotheses. By separately evaluating "novelty relative to a reference baseline," "diversity across constituent dimensions," and "predictive utility," the researchers can more clearly examine the trade-offs between these metrics—trade-offs that are often obscured when reduced to a single, aggregate score. Furthermore, the paper provides precise definitions for "novelty" and "value," carefully avoiding any implication that these terms are synonymous with absolute scientific originality or impact.
-
The metrics proposed in this work are grounded in clear operational definitions and supported by targeted validation methods. The "diversity kernel" is constructed to ensure positive semi-definiteness, with its limiting cases offering meaningful interpretations; meanwhile, the "value" metric is assessed by comparing predictive outcomes with and against the inclusion of a specific hypothesis. Through coverage calibration studies, controlled directional replication experiments, and the evaluation of treatment effects across different "reasoners," the paper provides a robust empirical foundation for the performance of these metrics. Compared to approaches that rely solely on unconstrained quality scores, this framework offers superior interpretability and auditability.
Critical weaknesses
-
The existing validation does not establish the reliability of this framework when it comes to evaluating "naturally generated and scientifically plausible" alternatives. Experiment E1 only validates the estimation of "unseen mass" within a selected reference population; it does not validate the effectiveness of the resulting "novelty" scores for evaluating alternative mechanisms. Experiment E2 assesses a collection of restated, previously published ideas. In Experiment E3, research hypotheses are constructed based on latent research findings; for those hypotheses deemed "supported," the authors explicitly verify whether they indeed lead to the predetermined correct answers (see Appendices D and F). While the resulting rankings serve as a useful sanity check, they do not provide sufficient evidence that the framework can effectively distinguish between competing scientific hypotheses—particularly those that are incomplete or indirect. Furthermore, although the study spans multiple disciplines, the human evaluation (concerning "novelty" and "diversity" metrics) is limited to AI-generated content. While I do not require these operational metrics to measure absolute scientific quality, substantiating the paper’s core contributions demands stronger evidence linking these metrics to their intended application—the evaluation of generated hypothesis sets.
-
Regarding the system comparisons, the paper lacks sufficient information on effective sample sizes and uncertainty. Section 6 and Tables 8–11 provide only point estimates for each system. However, the "diversity" metric employs filtering criteria specific to certain components, and the "value" metric applies a "headroom screen" tailored to specific reasoning models; moreover, cases with incomplete or ambiguous outputs were excluded from scoring (see Appendix C.5). The paper does not explicitly list the topics retained for each system and metric, nor does it clarify whether the comparisons are based on an identical set of topics. Appendix D mentions 17 overlapping topics between two groups of reasoning models, yet it fails to specify the total size or composition of each group. In the absence of these denominators, a summary of missing data, and estimates of pairwise uncertainty, it is difficult to disentangle the effects attributable to "workflow" from those resulting from differences in topic composition or sampling variation. This directly impacts the strength of the evidence supporting the "scaffold profiles" and the ranking results presented in the paper.
Questions for authors
-
How many topics were covered when assessing each system across the dimensions of "Novelty," "Diversity," and "Value"? Please specify the "headroom cohorts" defined for each specific reasoner, the data excluded due to incomplete runs or invalid responses, and whether the same intersection of topics was used when comparing "scaffolded" reasoning against "direct" reasoning. If the evaluation cohorts differ, do the primary conclusions still hold when restricted to a matched set of topics?
-
Could you provide paired topic-bootstrap confidence intervals for the differences between "scaffolded" and "direct" reasoning, and quantify the stability of the reported rankings? Additionally, please clarify the number of independent generation runs performed by each system for each topic, and specify which sources of variation are accounted for in your uncertainty estimates.
-
Beyond the manually defined A–D treatment conditions, are there independently evaluated samples of system outputs that demonstrate the reasonableness of these metrics—for instance, in cases involving plausible alternative mechanisms, scientifically incorrect mechanisms that rarely appear in reference materials, or hypotheses with predictive scopes extending beyond the specific "hidden-question set"? Providing cross-disciplinary examples along with expert evaluations would significantly strengthen my confidence in the intended use of this benchmark.
4 comments
Response to Question 2: Statistical Uncertainty, Repeated-Run Robustness, and Ranking Stability
We thank the reviewer for highlighting the importance of quantifying statistical uncertainty and assessing ranking stability. We have conducted 10,000 paired topic-level bootstrap resamples and three independent generation runs per system configuration. The detailed statistical results are also reported in our response to Reviewer CBef's Question 2.
(1) Bootstrap confidence intervals. We performed paired bootstrap resampling over identical evaluation cohorts across systems: 51 held-out topics for Novelty and Diversity, and 42 Gemini-eligible topics for Value.
The scaffold–Direct improvements remain positive for Novelty (Qwen: +0.079, 95% CI [0.043, 0.115]; GLM: +0.156, [0.104, 0.208]) and Value (Qwen: +0.031, [0.012, 0.050]; GLM: +0.040, [0.018, 0.062]).
For Mechanism Diversity, the results are backbone-dependent: Qwen shows a positive improvement (+1.168, [0.921, 1.423]), whereas GLM's confidence interval includes zero (−0.067, [−0.195, 0.061]). These findings support the positive aggregate Novelty and Value results while preserving the observed differences in Diversity.
(2) Repeated-run robustness. Across three independent generation runs, the mean scaffold–Direct improvements remain consistent with the original results:
- Novelty: Qwen +0.076 ± 0.012; GLM +0.151 ± 0.019.
- Value: Qwen +0.030 ± 0.008; GLM +0.038 ± 0.010.
- Mechanism Diversity: Qwen +1.253; GLM −0.005.
Thus, the positive Novelty and Value trends persist across repeated generations, while Mechanism Diversity remains substantially stronger for Qwen than for GLM.
(3) Ranking stability. We additionally compared the rankings of all 11 workflows within each generator backbone across the three runs. Across Novelty, Mechanism Diversity, and Value, the average pairwise Spearman correlation is 0.94, Kendall correlation is 0.85, and top-three overlap is 88.9%. The corresponding ranges are 0.91–0.96, 0.80–0.89, and 77.8%–100%, respectively.
These results indicate that the overall workflow rankings are highly consistent across independent runs, although individual pairwise ordering differences should not be interpreted as statistically significant without further testing.
Overall, the paired bootstrap, repeated-generation experiments, and ranking analysis provide complementary evidence for the robustness of our main findings. In particular, the positive aggregate Novelty and Value improvements are stable, whereas the Diversity results remain backbone-dependent. We will incorporate these uncertainty estimates and reproducibility analyses into the revised manuscript.
Response to Question 3: Independent Expert Validation and Challenging Scientific Hypotheses
We thank the reviewer for emphasizing the importance of evaluating naturally generated scientific hypotheses beyond the controlled A–D treatments. To address this concern, we provide independent domain-expert evaluations across AI/ML and Life Sciences, together with two complementary real-generation case studies involving plausible alternative mechanisms and scientifically questionable but novel hypotheses.
(1) Independent expert validation. As reported in our additional experiments, we evaluated 100 naturally generated hypotheses across AI/ML and Life Sciences, with two independent domain experts per field.
Table 2. Independent Expert Validation of Predictive Value
| Domain | Hypotheses | Expert Agreement ↑ | Value–Expert Spearman ρ ↑ |
|---|---|---|---|
| AI/ML | 50 | 0.82 | 0.71 |
| Life Sciences | 50 | 0.78 | 0.65 |
| Overall | 100 | 0.80 | 0.68 |
The positive correlations across both domains provide external evidence that IdeaPrism's Value metric aligns with independent expert assessments of naturally generated hypotheses, rather than merely distinguishing manually constructed treatment conditions.
To further validate all three dimensions, we conducted an expert evaluation on 240 naturally generated hypotheses from four AI-scientist systems across six topics in AI and Life Sciences. Each topic was independently assessed by two corresponding domain experts under blinded conditions.
Table 3. Agreement Between IdeaPrism and Independent Domain Experts
| Dimension | Metric | Human–Human | IdeaPrism–Human |
|---|---|---|---|
| Novelty | Pairwise Ranking Agreement ↑ | 86.2% | 80.4% |
| Diversity | Effective Direction Count MAE ↓ | 0.31 | 0.47 |
| Value | Predictive Informativeness Agreement ↑ | 83.5% | 76.8% |
These results demonstrate substantial alignment with independent expert judgments across Novelty, Diversity, and Value, complementing our earlier Value-focused validation and providing additional evidence for IdeaPrism's applicability to actual AI-scientist outputs.
(2) Case Study A: A Plausible Alternative Mechanism. We examined a hypothesis naturally generated by Virtual Scientist (Qwen2.5-72B) for the Lost in the Middle problem.
While the designated explanation emphasizes positional attention bias, the generated hypothesis proposes insufficient utilization of document structure and topical coherence as an alternative contributor to middle-position retrieval failures.
Conditioning on this hypothesis improves prediction accuracy from 7/12 to 11/12 (Value = +0.333), and independent domain experts assessed it as scientifically useful.
This case illustrates that IdeaPrism can recognize the predictive utility of plausible alternative mechanisms without requiring agreement with a predefined target explanation.
(3) Case Study B: A Novel but Scientifically Questionable Mechanism. We examined Weighted-Residual Grokking Theory, naturally generated by MultiAgent Ideator (GLM-4-32B-0414). The hypothesis claims that sufficiently influential residual pathways are necessary for grokking.
IdeaPrism assigns high Novelty (0.933) because the proposed mechanism is unmatched in the reference pool of 150 hypotheses, but negative Value (−0.250), with prediction accuracy decreasing from 10/12 to 7/12.
Importantly, Levi et al. (ICLR 2024) demonstrated analytically and numerically that grokking can occur in simple linear models without requiring residual or skip connections, challenging the hypothesis's universal necessity claim.
This case illustrates that high reference-relative novelty does not imply scientific correctness or predictive utility. IdeaPrism distinguishes these properties rather than conflating them into a single scientific-quality score.
(4) Predictive scope and incomplete hypotheses. We acknowledge that Value measures predictive utility over the evaluated hidden questions, potentially underestimating hypotheses with broader predictive implications. Similarly, incomplete scientific specifications are reflected in component-level Diversity coverage. We will clarify these limitations, particularly for incomplete or indirect hypotheses.
Overall, our cross-disciplinary expert evaluations and complementary real-generation cases provide evidence beyond the controlled A–D treatments. They demonstrate that IdeaPrism can recognize scientifically useful alternatives without target matching, distinguish novelty from predictive utility, and provide interpretable multidimensional assessments of naturally generated hypotheses.
Thank you again for reviewing our work. Before the discussion period closes, we would welcome any further questions or feedback and would be happy to clarify any points that remain unclear.
Response to Question 1: Effective Sample Sizes, Headroom Cohorts, Missing Data, and Matched-Topic Robustness
We thank the reviewer for raising this important concern. We agree that reporting effective sample sizes and evaluation cohorts explicitly is essential for interpreting system-level comparisons. We would like to clarify that all scaffolded and Direct LLM configurations were evaluated on identical final topic sets within each metric, rather than independently selected topic subsets.
(1) Effective sample sizes across evaluation dimensions.
All 22 configurations (11 workflows × 2 generator backbones) were evaluated on the same 51 held-out topics for Novelty and all four Diversity components (Mechanism, Intervention, Prediction, and Experiment).
For Diversity, hypotheses with unspecified required scientific attributes are excluded only from the corresponding component calculation. Importantly, this component-level filtering did not result in any topic-level exclusions: all 51 topics remained scorable for every workflow and Diversity component.
(2) Reasoner-specific headroom cohorts.
Value applies the reasoner-specific headroom criterion S−h≤0.85. Among the 51 held-out topics, Gemini-2.5-Flash retains 42 topics, while GPT-4.1 retains 21 topics, with 17 topics shared by both cohorts, consistent with the common-cohort analysis already reported in Appendix D.
Crucially, all Section 6 system comparisons use Gemini-2.5-Flash as the fixed Value reasoner. Thus, all 22 configurations, including Direct LLM, are evaluated on the same 42 Gemini-eligible topics. GPT-4.1 is used separately for E3 validation and does not affect the system-comparison cohort.
Table 1. Effective Evaluation Cohorts Across Dimensions
(3) Incomplete runs and invalid responses.
Appendix C.5 already documents our treatment of incomplete runs and ambiguous responses. We further clarify that no additional topic-level exclusions occurred due to incomplete generation runs or invalid Value responses. Consequently, the final effective denominators remain identical across workflows: 51 topics for Novelty and each Diversity component, and 42 topics for Value.
While incomplete scientific specifications affect the number of hypotheses contributing to individual Diversity components, they do not change the topic cohorts used for system comparisons. We will make the distinction between hypothesis-level filtering and topic-level eligibility more explicit and provide corresponding specification-coverage statistics and topic identifiers.
(4) Matched-topic comparisons and robustness of the main conclusions.
We emphasize that the evaluation cohorts differ across metrics, but not across systems within the same metric. All scaffolded and Direct LLM configurations use exactly the same 51 topics for Novelty and Diversity and the same 42 Gemini-eligible topics for Value.
Therefore, the original scaffold–Direct comparisons are already matched-topic comparisons. Restricting each comparison to the intersection of valid topics does not alter the evaluated topic sets or the reported numerical results.
Importantly, the main conclusions remain unchanged under matched-topic evaluation. Scaffolds exhibit positive aggregate improvements in Novelty (Qwen: +0.079; GLM: +0.156) and Value (Qwen: +0.031; GLM: +0.040), whereas Mechanism Diversity shows backbone-dependent behavior (Qwen: +1.168; GLM: −0.067).
These findings demonstrate that the observed scaffold profiles are not artifacts of differences in topic composition. Instead, they reflect distinct patterns of exploration novelty, scientific-direction diversity, and predictive utility across AI-scientist workflows.
Overall, the concern stems from insufficiently explicit reporting of effective denominators rather than unmatched evaluation cohorts. We will revise the manuscript to clearly document the complete headroom cohorts, topic-level exclusion counts, component-level specification coverage, and matched-topic evaluation protocol.