- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Causal Referee (CaRe), a framework that uses an LLM to coordinate multiple statistical causal discovery algorithms. Repeated resampling estimates each expert’s support for candidate relations. These supports are aggregated hierarchically at the skeleton, v-structure, and edge-orientation levels. For relations near the decision boundary, the LLM assigns relation-specific expert weights based on variable semantics, dataset characteristics, and the experts’ assumptions and opinions. Direction propagation, conflict checks, and acyclicity constraints produce the final DAG.
The evaluation covers five datasets sampled from benchmark Bayesian networks and three real-world or domain-derived datasets. The paper reports structural error, precision-recall AUC, and LLM query counts, with overall improvements over the evaluated individual algorithms and uniform aggregation. Additional experiments examine benchmark-identity masking, a newly generated graph, expert removal, the hierarchical design, query selection, and final DAG completion.
Critical strengths
The method provides a clear and implementable way to combine LLMs with statistical causal discovery. Statistical methods retain their own judgments, while the LLM changes their influence on selected disputed relations. Separating skeletons, v-structures, and orientations provides a common interface for heterogeneous graph outputs. Compared with asking an LLM to produce an entire graph, this design makes the sources of individual decisions easier to inspect.
Critical weaknesses
The evidence does not sufficiently isolate expert-reliability assessment as the source of improvement. The LLM sees each expert’s stance, and the prompts explicitly ask it to assess the causal plausibility of the candidate relation. It could therefore first decide whether the relation is plausible from semantics and then assign higher weights to experts that support that decision. Although expressed through weights, this need not differ from direct causal judgment constrained by the experts’ candidate relations. The direct causal expert in Table 2 predicts an entire graph, which differs from CaRe’s task of resolving individual disputed relations. Without a direct-relation baseline using the same disputed cases and context, and comparisons with global or layer-specific expert weights, the central interpretation of the LLM’s role remains insufficiently supported.
The evaluation does not fully characterize uncertainty in the complete pipeline, and graph conversion is insufficiently specified. Appendix D.3 fixes the data, expert outputs, and resampling support scores, then repeats only the LLM stage. This measures LLM variability but does not establish whether improvements persist across data realizations and expert estimation errors. Four sample sizes are not a substitute for independent repetitions. Moreover, DAGs, CPDAGs, and PAGs are converted into directed adjacency matrices without sufficiently explaining how undirected edges, circle endpoints, and bidirected edges are handled. These choices affect orientation errors and AUC, and therefore the interpretation of comparisons with statistical baselines.
Questions for authors
- Could the LLM directly select the relation state for the same disputed cases, using the same variable information and dataset context, followed by the same graph-construction and validity checks? How does this compare with CaRe? This would help establish whether reweighting is more reliable than direct relation judgment under matched conditions.
- Were multiple independent data seeds used at each sample size in the synthetic experiments? Could you report paired results and uncertainty after regenerating the data and rerunning the experts and resampling procedure, rather than repeating only the LLM stage with fixed expert outputs?
- How are DAGs, CPDAGs, and PAGs converted into directed adjacency matrices for evaluation? Is AUC computed from binary final graphs or continuous support scores? For Arctic-Sea, how is the mismatch between the cyclic/bidirectional reference graph and the DAG output space handled?