- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper examines whether commonly used causal discovery benchmark graphs agree with domain literature. The authors first survey benchmark usage in recent causal discovery papers, identifying 69 empirical benchmarks, and then select ten frequently used benchmarks for further analysis.
The proposed pipeline retrieves papers for variable pairs, extracts conditional independence relations from benchmark graphs, and uses an LLM to assess agreement between these relations and reported statistical findings. Approximately 35,000 papers are processed. The fraction of inconsistent results among answerable cases is used to measure disagreement with domain knowledge. A manual check of 100 randomly sampled papers reports 90% verifier accuracy. The main finding is that inconsistency rates vary substantially across benchmarks, with lower rates for Sachs and higher rates for Asia and Child.
Critical strengths
The paper addresses an important and often overlooked evaluation problem. Lower structural error against an empirical benchmark does not automatically imply better agreement with the real world. This distinction is particularly relevant for LLM methods that use variable semantics and may disagree with a fixed reference graph. The authors attempt to build a traceable literature-auditing workflow. The work combines a benchmark-usage survey, large-scale retrieval, conditional-relation checking, and manual validation. Requiring supporting passages from the source papers also facilitates later inspection. With a sounder core methodology, this resource could help benchmark maintenance.
Critical weaknesses
The graph-based extraction does not correctly and fully implement the required conditional independence reasoning. Section 3.1 interprets d-connection as correlation, but the Markov property only guarantees that d-separation implies conditional independence. The converse requires additional assumptions such as faithfulness, and statistical dependence need not imply correlation in its usual sense. More concretely, the appendix algorithm considers only collider-free paths and finds minimal sets blocking those paths. It does not check paths opened by conditioning on colliders or their descendants. It also immediately returns independence when no collider-free path exists. For example, in X → C ← Y, marginal independence does not imply independence conditional on C. This algorithm is insufficient for the arbitrary conditioning-set comparisons described in the main text. Checking only minimal separating sets also does not establish the relation for the conditioning set actually used in a paper. Since the inconsistency rates depend on these judgments, this issue could change the main findings.
Questions for authors
Is the appendix algorithm the implementation used for the reported results? If so, how are paths opened by conditioning on colliders or their descendants handled? Could all actual queries be checked using a complete d-separation procedure, with updated inconsistency rates and rankings?
How does the pipeline handle a paper whose conditioning set is not a minimal separating set? Is faithfulness assumed when interpreting d-connection? How does the verifier distinguish statistical dependence, correlation, predictive ability, and statistical nonsignificance?