- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces OpenHypBench (Open-ended Hypothesis Discovery), a synthetic benchmark for open-ended causal hypothesis discovery. The authors also introduce HypAgent, an LLM-based iterative search procedure that uses residualization, triage, and refinement.
Critical strengths
-
The problem formulation is interesting.
-
Evaluating proposals by intervening on a hidden SCM is sensible.
-
The observation that most misses happen after the relevant variable has already been identified is informative.
Critical weaknesses
-
The technical novelty is somewhat incremental. HypAgent is essentially OLS residualization, a univariate correlation-ratio screen, and LLM-guided refinement. I do not see a novel estimator or search objective.
-
The benchmark looks synthetic. Claude Opus 4.8 proposes the mechanisms, while all LLM-based methods use Claude Sonnet 4.6. Thus, performance may partly reflect shared priors.
-
The method relies on several ad hoc design choices (Appendix D.1). The ablation results are also reported without confidence intervals.
-
The baseline comparison seems unfair. Correlation mining, subgroup analysis, and PC were not meant to output full hypotheses with conditions and magnitudes.
-
The anonymous repository contains only a ReadMe file. I cannot check the benchmark construction, evaluator, or reported results.
Questions for authors
Please see the above weaknesses.
3 comments
Rebuttal contd.
W4: Statistical baselines
These comparisons are not intended to criticize the baselines at their original tasks. They test whether established data-analysis tools can be adapted to open-ended hypothesis discovery. We provide explicit output adapters (Appendix D.3), yet their associations, subgroups, and graph edges remain insufficient to recover complete hypotheses. This is precisely the capability gap our benchmark exposes that traditional ML baselines cannot solve hypothesis discovery in openended setting. The primary comparisons are against LLM and agentic methods.
W5: Repository
We apologize that the repository was incomplete when inspected. The repository was updated with the code and can be found here: https://anonymous.4open.science/r/ICLR-hyp-disc-272E/
We verified that the released pipeline runs end to end from benchmark construction through evaluation. The paper’s Reproducibility Statement points readers to the construction, evaluation, configuration, and baseline details in Section 4 and Appendices C–E. The repository contains the benchmark generator, evaluator, method and baseline configurations, prompts, and execution instructions.
Request to review rebuttal
Dear reviewer 4HDp, As we are nearing the end of author response period, we would be grateful if you could review our complete rebuttal and let us know of any remaining questions or points requiring clarification. If you find that your concerns have been satisfactorily addressed, we request that you consider updating your evaluation of the submission.
Rebuttal
We thank the reviewer for recognizing the value of the open-ended formulation and hidden-SCM evaluation.
W1: Technical novelty
The central novelty of our work is the task and its causal evaluation. The Introduction motivates the scientific stage missing from most existing settings, while Section 3, Definition 1 formalizes the problem: given only a dataset and outcome, a method must discover multiple mechanisms without a prespecified treatment, research question, functional form, condition, magnitude, or number of hypotheses.
HypAgent is our first solution to this new task. As described in Section 5 and Algorithm 1, accepted discoveries change the residual search target, triage selects directions in the remaining signal, and refinement converts those directions into complete, testable claims. We conducted additional experiments with another LLM backbone, Qwen3-32B, and found that HypAgent’s advantage holds:
Claude Code is tied to Claude and therefore has no Qwen counterpart. HypAgent remains the strongest method among methods with portable backbones.
W2: Synthetic construction and shared priors
OpenHypBench grounds schemas and empirical marginal distributions in public datasets while constructing hidden causal mechanisms that can be evaluated interventionally. We agree that the tasks are synthetic, but this is essential for two reasons:
Since we create the tasks and retain the hidden SCM, we can admit any arbitrary proposal and evaluate it accurately. This enables the verification of any hypothesis, making the setting open-ended.
Before constructing OpenHypBench, we investigated whether published findings over real-world datasets could provide discovery targets. We collected roughly 150 hypotheses from papers associated with 15 datasets and evaluated both HypAgent and LLM (no code). Both reproduced approximately 90%. Specifically, vanilla LLM without agentic capabilities did so using only the dataset and domain context, without executing additional analysis. Thus, high performance on such datasets would primarily measure retrieval or memorization rather than discovery from data. This result motivated using previously unseen mechanisms whose validity can be determined only through the hidden SCM. Moreover, our benchmark construction can be extended to any domain or dataset and does not rely on existing data as the data is generated for creation of the tasks. This ensures that LLMs cannot memorize it and evaluation is not contaminated.
The complete task list is reported in Appendix C, Table 5, and the scope of these synthetic hypotheses and tasks is acknowledged in Appendix A.
To test the influence of shared priors, we performed additional experiments in which we anonymized all feature names, renamed the outcome Y, and removed domain descriptions while preserving the data and SCMs:
Anonymization removes semantic names, not observational evidence: LLM (no code) still receives distribution summaries and 1,000 sampled rows. Its 0.25 F1 can therefore use visible sample patterns, while HypAgent’s 0.46 F1 reflects the additional benefit of systematic data exploration.
Second, we varied the benchmark constructor, HypAgent backbone, and recall judge across all 15 tasks:
Changing the judge alters F1 by at most 0.02, and replacing Claude in construction or discovery preserves performance. Thus, semantic names and Claude-family alignment do not explain HypAgent’s gains.
W3: Design choices and ablations
All settings in Appendices D.1–D.2 are fixed globally across the 15 tasks rather than tuned per task. Removing residualization, triage, or refinement substantially degrades performance, showing that each component contributes materially in the evaluated configuration. The ablations do not establish that every threshold is optimal. We ran the ablations three times for each task and report the median values. We have added this clarification to the paper.