- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies hybrid causal discovery methods (LLM + statistical methods) by introducing CausalMix, an evaluation framework that measures the causal graph recovery contributions of LLMs and numerical data separately. The framework evaluates causal discovery methods under different semantic conditions, data budgets, and data representations, with classical causal discovery methods as baselines. Additionally, the work introduces a causal discovery benchmark simulator and proposes a post-training procedure using SFT and GRPO for LLM-based causal discovery. The experiments cover several LLMs and causal graphs of different sizes. The results show that LLMs can recover considerable graph structures based on variable names alone, while adding numerical data does not consistently improve their performances. The post-training procedure also improves output validity.
Critical strengths
- The paper provides a systematic evaluation framework for hybrid causal discovery methods, allowing examining the performances contributed by semantic and numerical evidence separately.
- The framework includes different data representations and interventional data, which could be helpful for understanding the limitations of existing methods and developing more effective hybrid causal discovery approaches.
Critical weaknesses
- The paper's novelty and overall contribution are limited. The definition of hybrid causal discovery is straightforward, and the idea of evaluating the contributions of semantic and numerical evidence separately has been explored in existing works. I acknowledge that the proposed evaluation framework provides a more systematic combination of these components, however, the underlying idea is not fundamentally new. Furthermore, fine-tuning LLMs for causal discovery has also been studied in previous works. Although the paper introduces GRPO, its application to graph recovery does not appear to provide significant methodological novelty.
- The experiments may not sufficiently reflect practical causal discovery tasks. Although the paper evaluates graphs with up to 50 variables, many experiments still rely on old benchmarks, especially Asia, Earthquake, and Cancer. These datasets are relatively small, and the LLMs may have memorised their structures. More importantly, some of those old benchmarks contain knowledge misaligned with the domains, needing further discussion of their fitness for evaluating LLM-based methods. The larger-scale experiments provide additional results, but the synthetic graph configurations remain limited. I believe more experiments on new causal graphs would be helpful to demonstrate the generality and usefulness of the evaluation framework.
- The effectiveness of the proposed post-training procedure is not sufficiently demonstrated. The experiments show improvements in output validity, but the improvements in graph recovery and numerical evidence usage are limited. For instance, on Sachs, adding numerical data to the SFT+GRPO model provides almost no F1 improvement with original variable names. Moreover, the training graphs overlap with several evaluation graphs, while the held-out-family experiments only evaluate SFT rather than GRPO. The comparisons between SFT and GRPO also do not fully isolate the contribution of GRPO. Therefore, it remains unclear whether the proposed training procedure can significantly improve LLMs' causal discovery capabilities.
- Another concern is whether the experiments sufficiently demonstrate the usefulness of the evaluation framework for hybrid causal discovery. In practice, numerical evidence may be unreliable due to data quality, where semantic knowledge could help correct the results. However, the experiments mainly consider data generated from known Bayesian networks under strong assumptions. I believe more experiments under such conditions would be helpful to demonstrate the usefulness of the framework for evaluating the complementary strengths of semantic and statistical evidence.
Questions for authors
See the weakness part.
Withdrawal of Submission
Dear Reviewers and Area Chair,
We have decided to withdraw our submission. Thank you for the time and effort you spent reviewing our work. We appreciate your constructive feedback and will use it to improve the paper.
Best regards, The Authors