- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CauTion, which combines statistical causal discovery algorithms with an LLM and estimates their reliability without human annotations. The method first accepts decisions on which PC, GES, and CaMML fully agree, leaving the remaining variable pairs for arbitration. Consensus pseudo-labels estimate LLM accuracy, while leave-one-out agreement estimates algorithm accuracy. Separate trust weights are computed for edge existence and direction. The LLM is queried only when its answer can change the weighted voting decision. A final repair procedure reverses or removes edges to eliminate cycles.
Experiments cover six bnlearn networks with five independently sampled datasets per network, comparing statistical and LLM-augmented methods. The paper also includes component ablations, comparisons across six LLM backends, and evaluation on a newly constructed Airport network. CauTion achieves the best or tied-best mean SHD and F1 in the main experiments, with particularly clear gains on larger networks.
Critical strengths
The method provides a concrete mechanism for limiting LLM errors. CauTion estimates reliability separately for existence and direction and queries the LLM only when its answer can affect the decision. The query threshold follows directly from the weighted voting rule, giving selective invocation a clear decision-based justification.
Critical weaknesses
The assumptions underlying the reliability estimates are not sufficiently validated. Section 3.3.2 assumes that LLM reliability transfers from consensus to disputed pairs and argues that sharing a domain and variable space supports this assumption. However, disputed pairs are selected through algorithm disagreement and may concentrate on harder, semantically ambiguous, or more complex relations. Reweighting existence classes adjusts class proportions but does not address within-class difficulty shifts. Likewise, leave-one-out evaluation measures agreement rather than correctness. With only three algorithms, shared biases can produce high agreement despite incorrect decisions. Taking the mean of the higher-performing half of these proxy estimates does not establish the ensemble’s actual accuracy. These quantities determine both trust weights and query selection, so this concern directly affects the central contribution.
Questions for authors
Using the known reference graphs for evaluation, could you report LLM existence and direction accuracy separately on consensus and disputed pairs? How large is the gap after class reweighting? How closely do leave-one-out estimates track individual-algorithm and ensemble accuracy on disputed pairs?
With algorithm outputs, LLM answers, and cycle repair held fixed, how do estimated trust weights compare with fixed weights? As a diagnostic upper bound, how do results change when reliability is computed using the reference graph?
Rebuttal Summary
Dear Area Chair,
We sincerely thank you and the reviewers for reading our manuscript so carefully and for offering constructive suggestions that helped us present our method more clearly, evaluate it more rigorously, and better define its scope.
To assist your assessment, we provide a brief summary of our work.
Paper overview
LLM domain knowledge can help resolve causal relationships that observational data leave ambiguous, but an incorrect LLM judgment can introduce a spurious edge or reverse its direction. A central challenge in LLM-augmented causal discovery is therefore deciding when to trust LLM knowledge and when to rely on statistical evidence without access to the true graph. We propose CauTion, an ensemble framework that uses agreement among statistical algorithms to estimate the relative reliability of the LLM and the algorithm ensemble separately for edge existence and direction. CauTion directly accepts decisions on which all the algorithms agree and queries the LLM on disputed pairs only when its judgments are estimated reliable. On each of six benchmarks, CauTion achieves the best or tied-best mean SHD and F1 among the methods compared.
Discussion with reviewers
Recognized Strengths by Reviewers
Effective reliability calibration. Reviewer vpAi described the calibration scheme as “an effective way to use existing information” to compare algorithms without collecting more data. Reviewer Skpt similarly recognized “an effective approach” for combining statistical methods with LLM judgments on uncertain variable pairs.
Principled mitigation of LLM errors. Reviewer oVen recognized “a concrete mechanism for limiting LLM errors” and noted that the query threshold gives selective invocation “a clear decision-based justification”.
Strong empirical performance. Reviewer vpAi noted that “improvements hold across multiple base LLMs,” while Reviewer Skpt highlighted that the approach “achieves high accuracy compared to the baselines”.
Clear presentation of calibration and experiments. Reviewer vpAi described both the calibration scheme and the experiments as “clearly explained” and appreciated that the ablation and robustness sections “clearly separate components of the algorithm”.
Main Concerns and Questions Addressed
We hope that our detailed responses have carefully addressed all concerns, and that these efforts will be taken into consideration in the final assessment. We are truly grateful for your time and expertise throughout this review process.
Best regards,
The Authors of Submission #61821