acceptodds
Under review as a conference paper at ICLR 2027

Searching for What Matters: Open-ended Discovery of Multiple Hypothesis with Agentic Data Exploration

Abstract

Discovering what drives an outcome from data is central to science and often requires deciding what is worth investigating before a specific hypothesis is known. Yet most existing hypothesis-generation settings substantially narrow this search by providing a research question, target relationship, or curated context. We introduce Open-ended Hypothesis Discovery, where a system receives only a large tabular dataset and an outcome and must discover multiple data-supported hypotheses without knowing in advance the relevant variables, functional forms, conditions, or number of underlying mechanisms. To study this problem, we introduce OpenHypBench, a benchmark built from structural causal models grounded in real-world data schemas, containing 351 planted hypotheses across 15 tasks and five domains, with 53-65 variables and 1M-10M rows per task. The causal model is hidden from the discovery method and retained only by the evaluator, allowing proposed hypotheses to be tested through interventions and separating observational discovery from causal validation. We further propose HypAgent, which treats the dataset as an interactive environment and performs iterative search by proposing candidate directions, triaging and refining promising hypotheses, and removing explained signal so that each accepted discovery redirects subsequent search. HypAgent achieves 0.47 precision, 0.51 recall, and 0.49 F1 score compared with 0.35 for the strongest baseline. Our analysis shows that many failures occur after a relevant variable has already been surfaced, suggesting that correctly characterizing a mechanism's functional form and conditions, rather than merely identifying relevant variables, remains a key challenge in open-ended discovery.

Primary area:
datasets and benchmarks
Keywords:
Hypothesis discovery, openendedness, hypothesis generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4HDpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces OpenHypBench (Open-ended Hypothesis Discovery), a synthetic benchmark for open-ended causal hypothesis discovery. The authors also introduce HypAgent, an LLM-based iterative search procedure that uses residualization, triage, and refinement.

Critical strengths
  1. The problem formulation is interesting.

  2. Evaluating proposals by intervening on a hidden SCM is sensible.

  3. The observation that most misses happen after the relevant variable has already been identified is informative.

Critical weaknesses
  1. The technical novelty is somewhat incremental. HypAgent is essentially OLS residualization, a univariate correlation-ratio screen, and LLM-guided refinement. I do not see a novel estimator or search objective.

  2. The benchmark looks synthetic. Claude Opus 4.8 proposes the mechanisms, while all LLM-based methods use Claude Sonnet 4.6. Thus, performance may partly reflect shared priors.

  3. The method relies on several ad hoc design choices (Appendix D.1). The ablation results are also reported without confidence intervals.

  4. The baseline comparison seems unfair. Correlation mining, subgroup analysis, and PC were not meant to output full hypotheses with conditions and magnitudes.

  5. The anonymous repository contains only a ReadMe file. I cannot check the benchmark construction, evaluator, or reported results.

Questions for authors

Please see the above weaknesses.

3 comments

We thank the reviewer for recognizing the value of the open-ended formulation and hidden-SCM evaluation.

W1: Technical novelty

The central novelty of our work is the task and its causal evaluation. The Introduction motivates the scientific stage missing from most existing settings, while Section 3, Definition 1 formalizes the problem: given only a dataset and outcome, a method must discover multiple mechanisms without a prespecified treatment, research question, functional form, condition, magnitude, or number of hypotheses.

HypAgent is our first solution to this new task. As described in Section 5 and Algorithm 1, accepted discoveries change the residual search target, triage selects directions in the remaining signal, and refinement converts those directions into complete, testable claims. We conducted additional experiments with another LLM backbone, Qwen3-32B, and found that HypAgent’s advantage holds:

MethodClaude F1Qwen F1
XGBoost+LLM0.300.30
LLM (no code)0.240.18
ReAct0.110.07
AutoDiscovery0.270.19
HypAgent0.490.46

Claude Code is tied to Claude and therefore has no Qwen counterpart. HypAgent remains the strongest method among methods with portable backbones.

W2: Synthetic construction and shared priors

OpenHypBench grounds schemas and empirical marginal distributions in public datasets while constructing hidden causal mechanisms that can be evaluated interventionally. We agree that the tasks are synthetic, but this is essential for two reasons:

  1. Since we create the tasks and retain the hidden SCM, we can admit any arbitrary proposal and evaluate it accurately. This enables the verification of any hypothesis, making the setting open-ended.

  2. Before constructing OpenHypBench, we investigated whether published findings over real-world datasets could provide discovery targets. We collected roughly 150 hypotheses from papers associated with 15 datasets and evaluated both HypAgent and LLM (no code). Both reproduced approximately 90%. Specifically, vanilla LLM without agentic capabilities did so using only the dataset and domain context, without executing additional analysis. Thus, high performance on such datasets would primarily measure retrieval or memorization rather than discovery from data. This result motivated using previously unseen mechanisms whose validity can be determined only through the hidden SCM. Moreover, our benchmark construction can be extended to any domain or dataset and does not rely on existing data as the data is generated for creation of the tasks. This ensures that LLMs cannot memorize it and evaluation is not contaminated.

The complete task list is reported in Appendix C, Table 5, and the scope of these synthetic hypotheses and tasks is acknowledged in Appendix A.

To test the influence of shared priors, we performed additional experiments in which we anonymized all feature names, renamed the outcome YY, and removed domain descriptions while preserving the data and SCMs:

MethodOriginal F1Anonymized F1
LLM (no code)0.240.25
HypAgent0.490.46

Anonymization removes semantic names, not observational evidence: LLM (no code) still receives distribution summaries and 1,000 sampled rows. Its 0.25 F1 can therefore use visible sample patterns, while HypAgent’s 0.46 F1 reflects the additional benefit of systematic data exploration.

Second, we varied the benchmark constructor, HypAgent backbone, and recall judge across all 15 tasks:

ConstructorBackboneJudgeF1
Opus 4.8Sonnet 4.6Opus 4.80.49
Opus 4.8Sonnet 4.6GPT-50.49
Opus 4.8Qwen3-32BOpus 4.80.46
Opus 4.8Qwen3-32BGPT-50.45
GPT-5Sonnet 4.6Opus 4.80.51
GPT-5Sonnet 4.6GPT-50.52
GPT-5Qwen3-32BOpus 4.80.49
GPT-5Qwen3-32BGPT-50.51

Changing the judge alters F1 by at most 0.02, and replacing Claude in construction or discovery preserves performance. Thus, semantic names and Claude-family alignment do not explain HypAgent’s gains.

W3: Design choices and ablations

All settings in Appendices D.1–D.2 are fixed globally across the 15 tasks rather than tuned per task. Removing residualization, triage, or refinement substantially degrades performance, showing that each component contributes materially in the evaluated configuration. The ablations do not establish that every threshold is optimal. We ran the ablations three times for each task and report the median values. We have added this clarification to the paper.

AuthorsOpenReview ↗

Rebuttal contd.

W4: Statistical baselines

These comparisons are not intended to criticize the baselines at their original tasks. They test whether established data-analysis tools can be adapted to open-ended hypothesis discovery. We provide explicit output adapters (Appendix D.3), yet their associations, subgroups, and graph edges remain insufficient to recover complete hypotheses. This is precisely the capability gap our benchmark exposes that traditional ML baselines cannot solve hypothesis discovery in openended setting. The primary comparisons are against LLM and agentic methods.

W5: Repository

We apologize that the repository was incomplete when inspected. The repository was updated with the code and can be found here: https://anonymous.4open.science/r/ICLR-hyp-disc-272E/

We verified that the released pipeline runs end to end from benchmark construction through evaluation. The paper’s Reproducibility Statement points readers to the construction, evaluation, configuration, and baseline details in Section 4 and Appendices C–E. The repository contains the benchmark generator, evaluator, method and baseline configurations, prompts, and execution instructions.

AuthorsOpenReview ↗

Request to review rebuttal

Dear reviewer 4HDp, As we are nearing the end of author response period, we would be grateful if you could review our complete rebuttal and let us know of any remaining questions or points requiring clarification. If you find that your concerns have been satisfactorily addressed, we request that you consider updating your evaluation of the submission.

Reviewer_pbA5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper defines Open-ended Hypothesis Discovery: given only a large tabular dataset and a binary outcome, a system must return a set of hypotheses, each with variables, structural form, direction, condition, and absolute effect magnitude. It introduces OpenHypBench, 15 tasks in 5 domains with 1M-10M rows, 53-65 features, and 351 planted mechanisms from 7 functional-form families. Claude Opus 4.8 proposes the mechanisms, which are compiled into one SCM per task. Precision is scored by intervention on the hidden SCM, and recall uses a heuristic shortlist followed by a Claude Opus 4.8 equivalence judgment. The proposed HypAgent is a residual-driven loop: an LLM proposes directions, a correlation-ratio triage on 1,000 rows filters them, LLM code refinement tests them, and accepted hypotheses are removed from the outcome by OLS residualization. HypAgent reports P/R/F1 = 0.47/0.51/0.49 vs 0.31/0.39/0.35 for the best baseline, Claude Code. The paper also reports per-form recall, scaling studies and an ablation.

Critical strengths
  1. The task is well posed. Scoring proposals by intervention on a hidden SCM (Sec. 4.2) separates observational discovery from causal validation and gives precision credit to valid hypotheses outside the planted set.
  2. The datasets are large enough (1M-10M rows) that they cannot be pasted into context, and the benchmark includes hard mechanism families (Simpson's reversal, interaction reversal, mediation) plus decoy and null variables.
Critical weaknesses
  1. It is not established that the benchmark measures data-driven discovery rather than prior plausibility. The planted mechanisms are generated to "be plausible in the supplied domain" (App. E.1), and the column names carry domain meaning. A baseline that cannot run any analysis (LLM no code, App. D.3) reaches recall 0.33 vs 0.51 for HypAgent, and 0.28 even on Simpson's reversals. So a large part of the score may come from guessing plausible mechanisms from the schema. Without a control that removes this shortcut (e.g., anonymized column names), the benchmark does not isolate the capability it claims to measure.
  2. The same model family generates the mechanisms, judges recall, and runs all methods. Claude Opus 4.8 proposes the mechanisms and makes the final recall decision, and all LLM-based methods use Claude Sonnet 4.6. Shared priors across these roles could inflate scores and affect the ranking of methods. The authors leave cross-family evaluation to future work (App. A), but the benchmark's validity depends on it.
  3. The evaluation criterion is underspecified for the hard cases. A magnitude is correct if it "differs by at most 10%" from the interventional effect, but the paper does not say whether this is relative or absolute. With effects around 0.02-0.06, an absolute tolerance would accept almost anything. It is also unclear what a single scalar magnitude means for non-monotone, interaction-reversal or Simpson's mechanisms. Similarly, HypAgent's acceptance checks (Eqs. 4-6) rely on a linear coefficient, and the paper does not explain how they validate non-linear forms.
  4. Generality of the method and of the conclusions is not shown. HypAgent is co-designed with the benchmark's mechanism families and is not tested on any external benchmark (e.g., DiscoveryBench, BLADE) or on real data with known findings. Results appear to come from a single run with temperature 0.5-0.7. The RQ3 conclusion that "the search space matters more than the amount of evidence" is not well supported by Fig. 5, where the shared default configuration gives different F1 values across panels and the row and feature axes span very different ranges.
Questions for authors
  1. How does LLM (no code) perform when column names are anonymized or shuffled? How do the method rankings change with a non-Claude backbone and a non-Claude recall judge?
  2. In a setting where domain priors alone already recover many mechanisms, how do you define and measure "discovery"? Should the benchmark score only hypotheses that are not predictable from the schema?
  3. Is the 10% magnitude tolerance relative or absolute, and how is magnitude defined and matched for non-monotone, interaction-reversal and Simpson's mechanisms?
  4. OLS residualization removes accepted hypotheses linearly. How does this interact with overlapping or interacting mechanisms, where removing one effect may hide or distort another?
6 comments

We thank the reviewer for identifying controls needed to distinguish data-driven discovery from semantic plausibility and shared model priors.

W1 / Q1: Semantic plausibility versus data-driven discovery

The paper distinguishes plausibility from causal support: Section 4.2 defines precision using interventions on the hidden SCM, and Appendix D states that a proposal contributes to recall only after receiving interventional support. However, we agree that quantifying the contribution of semantic column names would resolve this doubt.

We therefore ran the proposed anonymization control on all 15 tasks. Every feature was renamed X1, X2, ..., Xn, the outcome was renamed Y, and the domain description was removed. Data values, types, SCMs, planted mechanisms, proposal budgets, and evaluation remained unchanged.

MethodOriginal P/R/F1Anonymized P/R/F1
LLM (no code)0.18 / 0.33 / 0.240.22 / 0.32 / 0.25
HypAgent0.47 / 0.51 / 0.490.45 / 0.46 / 0.46

The no-code baseline is not data-free: it receives distribution summaries of the entire dataset and 1,000 sampled observations but cannot execute further analyses. HypAgent retains 0.46 F1 versus 0.25 for this baseline after explicit semantic information is removed. Its advantage therefore persists without meaningful column names or domain descriptions.

W1 / Q2: What counts as discovery?

We do not define discovery as prior-free generation. Rather, we believe LLMs are more efficient at searching the hypothesis space, as priors can suggest plausible directions, but data is needed to establish whether they hold and to determine their form, conditions, direction, and magnitude.

Our metrics measure interventional correctness and coverage, not novelty relative to model pretraining. Plausible proposals receive no credit unless their complete claims are supported by intervention. The anonymization experiment isolates reliance on explicit schema semantics while preserving the observational evidence available to both methods.

W2 / Q1: Shared constructor, backbone, and judge

The original configuration is documented in Section 4.1, Appendix C.1, Appendix D, and Appendix D.2: Opus 4.8 constructs candidate benchmark mechanisms and performs recall-equivalence judgments, while Sonnet 4.6 is the discovery backbone. The submitted paper acknowledged cross-family evaluation as future work in Appendix A.

We have now independently varied the benchmark constructor, HypAgent backbone, and recall judge while keeping tools, budgets, and interventional scoring fixed. All results cover 15 tasks.

ConstructorHypAgent backboneRecall judgePRF1
Opus 4.8Sonnet 4.6Opus 4.80.470.510.49
Opus 4.8Sonnet 4.6GPT-50.470.510.49
Opus 4.8Qwen3-32BOpus 4.80.440.500.46
Opus 4.8Qwen3-32BGPT-50.440.490.45
GPT-5Sonnet 4.6Opus 4.80.490.530.51
GPT-5Sonnet 4.6GPT-50.490.550.52
GPT-5Qwen3-32BOpus 4.80.470.510.49
GPT-5Qwen3-32BGPT-50.470.540.51

Changing only the recall judge changes F1 by at most 0.02. Replacing Sonnet with Qwen as the hypothesis-generation backbone changes F1 by 0.01–0.04 across matched constructor and judge settings.

The method ranking is unchanged under Qwen:

MethodClaude P/R/F1Qwen P/R/F1
XGBoost+LLM0.20 / 0.40 / 0.300.25 / 0.38 / 0.30
LLM (no code)0.18 / 0.33 / 0.240.14 / 0.26 / 0.18
ReAct0.09 / 0.16 / 0.110.05 / 0.10 / 0.07
AutoDiscovery0.21 / 0.38 / 0.270.14 / 0.28 / 0.19
HypAgent0.47 / 0.51 / 0.490.44 / 0.50 / 0.46

The Qwen comparison covers all methods with LLM backbones and preserves their ranking. Claude Code is tied to Claude and therefore has no Qwen counterpart. Changing both the constructor and judge changes HypAgent’s F1 from 0.49 to 0.52 with Sonnet and from 0.46 to 0.51 with Qwen. These results have been added to Appendix D.4. The baseline ranking under the GPT-5 recall judge is also unchanged from the Claude judge.

W3 / Q3: Magnitude tolerance

The 10% tolerance is relative:

∣mh−mh⋆∣∣mh⋆∣≤0.10,\frac{|m_h-m_h^\star|}{|m_h^\star|}\leq 0.10,

where mhm_h is the claimed probability change and mh⋆m_h^\star is the realized contrast under the proposal’s experiment on the final SCM.

Magnitude is checked only after the required response structure is observed. Non-monotone mechanisms require the designated directional change across regions, interaction reversals require opposite effects across moderator values, and Simpson’s mechanisms require opposite marginal and within-strata associations. The revised paper makes the formula explicit in Section 4.2 and documents the operational response patterns below Table 4.

AuthorsOpenReview ↗

Rebuttal contd.

W3 / Q4: Overlapping mechanisms

Residualization jointly fits the encoded variables associated with all accepted hypotheses and overlapping hypotheses are not subtracted independently. It is a HypAgent specific approach for accounting for how much data is explained and helps budget which data to focus on for discovery. Since this is an openended setting, the number of hypotheses is unknown, and it is important for any method to systematically search the hypothesis space.

We agree that misspecification may suppress the remaining signal or distort an overlapping mechanism, thereby reducing recall. This limitation has been added to Appendix A. However, this cannot make an unsupported hypothesis pass because final scoring independently intervenes on the original SCM.

W4a: Generality and co-development with the benchmark

The primary contributions of the paper are introducing the task of openended hypothesis discovery and presenting a benchmark that addresses the evaluation limitations of openended hypothesis discovery. With workflows and execution becoming increasingly automated, generating correct hypotheses to test is increasingly important. Our paper introduces and formalizes this task rather than co-developing an agent for a benchmark we propose. HypAgent is our solution to this task and while it surpasses the existing methods, there is still scope for considerable improvement.

Finally, HypAgent is not given the benchmark’s seven mechanism families or their labels. As stated in Appendix D.1.3, refinement is not provided with a fixed menu of functional forms. HypAgent independently formulates the variables, relationships, conditions, and magnitude from its analysis of the data.

W4b: Existing external benchmarks

Existing benchmarks, including DiscoveryBench and BLADE, do not test openended hypothesis discovery, as they generally provide a research question, target relationship, treatment, curated context, or fixed answer set. These datasets are also prone to memorization as they have fixed questions and answer reference sets. They therefore evaluate hypothesis generation or analysis after the discovery target has already been specified. Moreover, observational real data benchmarks cannot interventionally evaluate an arbitrary new hypothesis, distinguish a false causal claim from a predictive association, or give credit to valid hypotheses outside a predefined reference set.

We had addtionally run experiments with real world datastes. In this setting, we use the 15 datasets used in our paper as the base datasets and then curated 10 hypotheses from the papers that cite these datasets and present some conclusion from the data. Under this setting, both HypAgent and vanilla LLM without any agentic capabilites reproduced approximately 90% of these hypotheses from the dataset and domain context. Sepcifically, vanilla LLM did so without executing additional data analysis. This indicates that evaluation on published findings can conflate discovery with retrieval or memorization. This is the core issue with evaluating openended hypothesis discovery from existing real world static datasets that our paper addresses through a synthetic SCM based benchmark that can evaluate arbitrary proposals. Moreover, our benchmark construction can be extended to any domain or dataset and does not rely on existing data as the data is generated for creation of the tasks. This ensures that LLMs cannot memorize it and evaluation is not contaminated.

W4c: Results appear to come from a single run at temperatures 0.5–0.7

Every method was run three times on each of the 15 tasks, and the reported results average these runs. Section 6 reports 95% confidence intervals from 2,000 cluster-bootstrap resamples over tasks, while Appendix D.2 reports the temperatures used by HypAgent.

W4d: Scaling conclusion

The scaling default is 1M rows, 50 features, and 21 planted mechanisms, with three from each archetype. Each experiment varies one dimension while holding the others fixed.

Increasing rows improves HypAgent from 0.44 to 0.61 F1, increasing features from 50 to 100 reduces it from 0.61 to 0.45, and increasing planted mechanisms does not cause a comparable decline. The conclusion follows from these directional trends, not from comparing one feature with one row.

Thank you for adding the anonymization experiment. It helps address some of my concerns about reliance on column names and domain knowledge. I still have one follow-up question: if we keep the column names and domain context the same but remove a mechanism from the data-generating process, would the system be less likely to propose that hypothesis? Do you have any results showing this? This would help clarify whether the system adjusts its conclusions based on the data, rather than mainly relying on what seems plausible in the domain.

Reviewer_DstMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates open-ended hypothesis discovery, where a system is tasked with generating multiple hypotheses from a large observational tabular dataset and a designated outcome, without prior knowledge of the relevant variables, functional forms, conditions, or number of underlying mechanisms. To study this problem, the authors introduce OpenHypBench, a benchmark comprising 351 planted mechanisms across 15 tasks spanning five domains. Each task contains 53–65 features and 1M–10M observations. The benchmark separates hypothesis generation from evaluation by using observational data for discovery and interventions on a hidden structural causal model for validation. Evaluation considers both the interventional support for generated hypotheses and the extent to which they recover the planted mechanisms.

Critical strengths

a. The benchmark uses a hidden SCM to evaluate interventional support rather than relying solely on textual similarity to reference hypotheses. It also gives credit to valid hypotheses beyond the planted mechanisms, making the evaluation more suitable for open-ended discovery.

b. HypAgent integrates candidate generation, screening, refinement, and residual-based search into an iterative framework. The ablation results demonstrate the contributions of individual components, particularly refinement, to the overall performance.

Critical weaknesses

a. It would be helpful to provide more details on how overlapping mechanisms are composed, how effect magnitudes are calibrated, and how natural-language hypotheses are translated into intervention-based evaluations. Additional validation showing that the planted mechanisms can be reliably recovered in the final SCM would further strengthen confidence in the benchmark.

b. The method estimates effect magnitudes using observational contrasts while evaluating hypotheses through interventions, and the relationship between these two could be clarified. Additionally, it would be useful to discuss how the acceptance criteria handle potential direction inconsistencies and whether the linear adjustment procedure is suitable for mediation and non-monotone mechanisms.

Questions for authors

a. Could the authors provide more details on how overlapping mechanisms are composed and their effect magnitudes are calibrated?

b. Could the authors clarify how natural-language hypotheses are translated into specific interventions and evaluation criteria? A few concrete examples, particularly for mediation and Simpson’s reversal, would help illustrate how the evaluator validates the underlying mechanisms.

c. Could the authors clarify the relationship between the observational effect estimates in Eq. (3) and the adjusted effects used for validation in Eqs. (4)–(6)? It would also be helpful to explain how the acceptance criteria handle direction inconsistencies, mediation, and non-monotone effects.

4 comments

We thank the reviewer for recognizing the value of the hidden-SCM evaluation and HypAgent’s iterative framework.

W1 / Q1: SCM composition and effect calibration

Thank you for raising this point for discussion. Each hypothesis is represented as a DAG and composed into a shared task-level SCM. Shared variables are represented by shared nodes, with their diverging children representing the different hypotheses and their effects. We ensure that the final SCM has no cycles. Only after this is confirmed do we generate the data in topological order, where the root nodes borrow their distributions from real-world datasets and the hypothesis DAGs provide the transformation to generate the values of their children from their parent nodes.

Ground-truth magnitude is the realized interventional difference in the hidden SCM, not the nominal construction coefficient. After composition, we execute every hypothesis' designated intervention and verify that all 351 planted mechanisms pass the post-composition recoverability check. This establishes consistency between the planted hypotheses and completed SCM. Refer to Section 4.1 and Appendix C.1 for more details.

W1 / Q2: Translating proposed hypotheses into interventions

Since only LLM-based methods return natural-language hypotheses, we prompt them to return structured outputs specifying the participating variables, direction, condition, and magnitude. These fields and the variable metadata determine the intervention. The hidden SCM is used to execute the resulting experiment. In our reported results, there were no parsing errors, and hence we are confident that each natural-language hypothesis was faithfully evaluated.

The hypothesis representation is defined in Section 3, Definition 1. Proposal normalization and evaluation are described in Section 4.2 (pp. 5–6) and Appendix D , while the seven interventional signatures are summarized in Appendix B, Table 4.

For example, the mediation hypothesis:

In the full population, intervening to increase BMI increases the probability of metabolic syndrome by 12 percentage points, with the effect mediated through elevated triglycerides.

The method returns the following structured fields:

FieldValue
Variables XhX_hBMI; elevated triglycerides
Direction ddIncrease
Condition ccFull population
Magnitude mhm_h0.12 (12 percentage points)

The claim text specifies that the effect is mediated through elevated triglycerides. During evaluation, this relationship is mapped to the mediation structure ψ\psi: BMI →\rightarrow triglycerides →\rightarrow metabolic syndrome.

The evaluator tests the response of the mediator to the BMI intervention and the response of the outcome to the mediator, corresponding to the mediation signature T→M→YT\rightarrow M\rightarrow Y in Table 4.

For Simpson’s reversal:

In the full population, intervening to assign cholesterol medication reduces the probability of metabolic syndrome by 8 percentage points, although medication has a positive marginal association with the outcome because diagnosed high cholesterol acts as a confounder.

The method returns the following structured fields:

FieldValue
Variables XhX_hCholesterol medication; diagnosed high cholesterol
Direction ddDecrease under intervention
Condition ccFull population
Magnitude mhm_h0.08 (8 percentage points)

The claim text describes a reversal between the marginal association and the interventional effect. During evaluation, this relationship is mapped to the Simpson’s-reversal structure ψ\psi.

The evaluator compares the marginal association with the interventional medication effect across diagnosis strata and requires the reversal specified for Simpson’s mechanisms in Table 4. These concrete examples have been added to Appendix D.

W2 / Q3: Observational screening versus causal validation

Equation 3 estimates the reported direction and magnitude from the original observational outcome. Equations 4–6 are HypAgent-specific internal mechanisms for direction agreement and persistence after adjustment. Mixed and non-monotone claims are checked within their stated regions rather than through one aggregate sign.

Evaluation is separate. The evaluator intervenes on the hidden SCM and compares the resulting response with the claimed hypothesis, as defined in Section 4.2 and Appendix D. Non-monotone, mediation, interaction-reversal, and Simpson’s claims are therefore evaluated through their corresponding interventional signatures in Table 4. An observationally accepted candidate can consequently be rejected by the evaluator if the claimed variables, conditions, direction, and magnitudes do not match with the interventional experiment.

AuthorsOpenReview ↗

Request to review rebuttal

Dear reviewer DstM, As we are nearing the end of author response period, we would be grateful if you could review our complete rebuttal and let us know of any remaining questions or points requiring clarification. If you find that your concerns have been satisfactorily addressed, we request that you consider updating your evaluation of the submission.

Thank you for the response. I still have some concerns.

The first is still about the evaluation. For mediation, does it check the mediated effect through the specified variable? Similarly, for Simpson's reversal, does it explicitly verify that the effect direction is reversed, rather than only checking the difference between observational and interventional estimates?

Another is about Eq. (5, 6), thank you for clarifying that Eqs. (4)–(6) are used for HypAgent's internal screening, while the final evaluation uses interventions. I have the question about whether adjusting for a mediator may remove part of the total causal effect, while a global linear adjustment may fail to capture non-monotone relationships.

For mediation, does it check the mediated effect through the specified variable?

Yes, for a hypothesis involving mediation. A correct proposed hypothesis should be able to recognize this. The evaluator compares this complete proposed hypothesis with the interventional evidence, so any missing or incorrect component causes the interventional experiment to deviate and lead to different experimental outcomes than those assumed by the incomplete hypothesis.

Similarly, for Simpson's reversal, does it explicitly verify that the effect direction is reversed, rather than only checking the difference between observational and interventional estimates?

Yes, since the difference between observational and interventional estimates is calculated in bins, the exact place where the direction is reversed is also checked through this. The evaluator explicitly requires the observational and interventional effects to have opposite directions, rather than merely differ in magnitude.

I have the question about whether adjusting for a mediator may remove part of the total causal effect, while a global linear adjustment may fail to capture non-monotone relationships.

Thank you for raising this question. HypAgent first refines a candidate into a structured mechanism, while accounting for variables to be mediators or different conditions that might enable the hypothesis. The confounder set ZZ in Eqs. (5)–(6) is then selected from the remaining covariates, excluding variables already assigned to the mechanism. Thus, for a mediation hypothesis T→M→YT \rightarrow M \rightarrow Y, HypAgent does not condition on MM and remove the indirect effect. Specifically, the adjustment tests whether the proposed mechanism survives accounting for separate candidate confounders or conditions.

Secondly, the global linear adjustment is not used to form a specific hypothesis but is one of the tests to check for confounders. Refinement phase examines the relationship across different regions of T⋆T^\star and analyses the corresponding region specific effects. This prevents opposing slopes from being averaged into a near zero global coefficient as we dont aggregate all the regions. Figure 4 further shows that HypAgent recovers non-monotone mechanisms in practice. We will clarify that Eqs. (5)–(6) describe the default confounder and robustness check, while hypotheses are refined by analyzing different effects of the proposed hypothesis aross different regions of data.

We would like to emphasize on the role of benchmark and the SCM here. The above questions are precisely why we use SCM so that we can evaluate such arbitrary complex hypotheses causally and faithfully. Prior benchmarks are prone to memorization and have fixed answer reference sets and hence cannot evaluate openended settings. Kindly let us know if you have further questions, we would be happy to answer them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.