- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The author proposed that in the causal structure learning algorithm based on the scoring function, the modification with the highest score does not necessarily correspond to the modification closest to the real causal graph. The paper presented an analysis method based on replaying search trajectories to measure the situation of the scoring function in each step of the candidate decisions. The author selected search trajectories that had actually been run, and re-enumerated all legal modifications at each previous decision point, respectively calculating the ranking of the scoring function and the accuracy rate of the real structure. The experimental results show that the scoring function is useful, but there are situations where the best candidate is not selected; re-arranging the candidates can significantly improve the accuracy rate; nearly half of the ranking losses come from candidates with the same score; complexity penalties cause sparse preferences. The author added LLM to the causal structure, and let LLM determine which one to choose in the next step among the candidate modifications that have been filtered by the scoring function. The conclusion drawn is that the model's advantage mainly comes from the semantics of the variables, rather than specific names in the benchmark data.
Critical strengths
- The author did not start from the outcome but rather focused on the candidate selection process during the analysis of the causal diagram, identifying the locations where the accuracy rate suffered.
- The author demonstrated that the ranking order of the candidate results and the scoring processing rules also have an impact on the experimental results.
Critical weaknesses
- The author mainly presented experimental evidence and mechanism analysis, but did not provide new theoretical proof, nor did they prove that LLM ranking is definitely superior to the original scoring.
- The experimental scale of the author was limited. The research was based on five discrete benchmark networks, among which the larger networks had 70 and 76 variables. Therefore, the experimental results cannot indicate that this phenomenon has the same effect in large-scale causal discovery tasks in the real world.
Questions for authors
- Does the LLM's benefit stem from true semantic understanding rather than familiarity with the benchmark data?
- Are the main findings of the paper statistically robust?
- How does the paper distinguish between the search ranking problem and the problem of the scoring function itself?
2 comments
Update (17 November): a second provider
A second provider. Since our reply, three Google models have run the same protocol, and they repeat every reading. The run was made on 17 November, after all the results above had been read. They ran both roles on the greenhouse and bike-share networks (27 cases each, names twice, five pseudonym maps) and the full role on child and insurance; the requests are those of the GPT runs except for the reasoning setting.
gemini-4-carbon and gemini-4-argon answer without reasoning, as gpt-5.6-sol does. With names they pick the truer direction at 97.4% and 98.7% of the greenhouse ties, against 78.2% for the implementation's tie-break, and at all 220 bike-share ties. With the variables renamed they are right at 51.9% and 50.9% on greenhouse, and at 43.5% and 39.6% on bike-share. Choosing at every step, they end 54 to 68 points above greedy with names and 38 to 63 points below it renamed, on both new networks and on child and insurance.
gemini-4-hydrogen cannot turn its reasoning off. With names it reads as the other two. Its renamed runs fail our audit gate: they parse 74% to 90% of their answers, and in the full role one presented position takes most of the choices. We report those runs and do not read them.
The finding of our reply therefore holds for four models from two providers: what the variable names convey is enough, and without the names the models do no better than chance. One caveat changes wording: both graphs were drafted with Claude (Anthropic), whose models we did not test. Appendix M.4 and its new table give every number; the traces are in the supplement.
Response to Reviewer VgA6
Thank you for the review.
Q1. Semantics or familiarity. The submission could not separate the two; the experiment we ran on 9 and 10 November does.
On public benchmarks the two cannot be told apart. Asked for each variable's parents with the network named, gpt-6.1-sol lists 117 of the 123 true edges, and the models name the benchmark in 5 of 6 probes from paraphrased descriptions that contain no variable name. So we wrote a network no model can have seen: 21 greenhouse variables and 30 edges, with the graph, the generator and the data checksums committed before the first model call.
The results closely match those on the benchmarks. With the real names, gpt-5.6-sol picks the truer direction at 94.7% of 151 ties, against 74.2% for the implementation's own tie-break. With the variables renamed it is right at 50.5% of 315. Choosing at every step, it recovers 89.5% of the reranking reference with names, 55.7 points above greedy [+30.4, +82.5], and falls 53.6 points below greedy when the variables are renamed. What the names convey is enough to produce the effect.
Recall still adds something on the benchmarks. There a model's parent list is nearly exact (precision 0.96 to 1.00 with the network named); on the new network the same model lists 22 of the 30 edges among 41 guesses. Its most common mistake on our network is to say that temperature drives the vents, as a thermostat would, where we drew the vents driving temperature.
We then repeated the experiment with a second model and a second network (a bike-share scheme, 24 variables). gpt-6.1-sol on the greenhouse network: 96.7% of 152 ties with names, 49.2% renamed. On the bike-share network both models are right at all 220 ties with names, and at 50.4% and 36.8% renamed.
Two caveats. Both graphs were drafted with an AI assistant from another provider and then corrected by us, in domains where common sense is a good guide. And on the greenhouse network we first ran 9 cases, then added 18 to narrow the intervals; Appendix M.4 reports the two batches separately. The submission read the description control as evidence against recall, and Appendix H withdraws that reading.
W1. Theory; whether LLM ranking is superior. The revision adds theory where the claims need it: Proposition 1 states when adding A→B and adding B→A tie under every score-equivalent score, Proposition 2 which of the two the likelihood floor prefers, and population scores remove sampling noise. Under population scores the tie choices alone decide the returned class in 166 of 199 faithful five-variable networks (Appendix A).
On superiority, the paper makes no such claim; the language model is one tie rule among several. At ties with a truer member on child and insurance, the frontier models choose it at 90.2% and 92.6%, against about 79% for the implementation's own tie-break, a lead that does not survive Holm correction (p = 0.094). On alarm the lead is 36.2 points and does survive it (p = 0.047).
W2. Scale. The study now covers eleven networks, the largest being pathfinder (109 variables), andes (223) and pigs (441), with 39 cases each. Resolving ties by the truth removes 41% of greedy's CPDAG SHD on andes and 59% on pigs, and covered-edge reversal recovers 94% and 91% of that; on pathfinder the tie loss is small and not significant (Table 1). On the real Sachs data resolving ties changes almost nothing (Appendix I).
Q2. Statistical robustness. Every interval resamples whole cases (10,000 bootstrap replicates). Contrasts that rest on nine cases are also tested with exact sign-flip tests, and families of contrasts are Holm-corrected (Appendix L). For example, against a coin flip, the implementation's tie-break is better in 8 of 9 cases on child and 9 of 9 on insurance, and worse in 8 of 9 on alarm (exact p = 0.027, 0.004, 0.012, all surviving Holm).
Q3. Search ranking or the score. Both, and the paper separates them. Every policy we compare takes its edit from the set the score accepts, so any difference between policies is a difference in ranking. What the score excludes is measured separately in Appendix C. From the default start, 54.9% of the single-edit oracle's F1 gain comes from edits the score accepts and 25.5% from edits it rejects under every setting we tried; from the empty start these are 80.5% and 7.3%. On 13 November we removed the score's estimation error altogether: with exact population scores on 2,206 small networks, resolving ties by the truth still leaves 18% of the ordering loss, which ranking by the truth removes. Part of the ranking problem therefore lies in the search itself (Appendix A).