- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Fisher Information Completion Search. The key idea is to use local Fisher information to identify parent sets and recover causal orderings without esimating high-dimensional densities. The authors establish theoretical guarantees for DAG recovery. Experiments on sythetic and real-world datasets demonstrate competitive performance.
Critical strengths
The paper introduces an interesting Fisher-information-based framework for sparse causal discovery, with theoretical recovery guarantees. The use of low-dimensional local information is particularly appealing for high-dimensional settings.
Critical weaknesses
Although FiCS demonstrates impressive performance in high-dimensional setting, the experimetns rely heavily on highly sparse DAGs with small maximus indegrees. Its nearly identical F1 scores accross different dimensions, together with its sensitivity to nonlinear mechanisms, raise concerns about whether the reported advantages generalize to more complex and realistic causal structures.
Questions for authors
- In Figure2, FiCS achieves remarkably similar F1 scores for p=50,100,200, even when p>n, while competing methods deteriorate substantially. Could the authors explain this phenomenon and provide addtional evidence supporting such dimensional robustness?
- The synthetic experimetns restrict the maximum indegree to two, matching the conditioning size q=2. How does FiCS perform on more complex DAGs with larger maximum indegrees?
- Appendix G.2 shows that FiCS peforms substantially better on linear mechanisms than on fully nonlinear causall discovery. Could the authors clarify the source of this sensitivity and its implications for nonlinear causal discovery?
4 comments
Response to Reviewer EHgh [2/2]
Question 2. Graphs with larger maximum indegree
The synthetic experimetns restrict the maximum indegree to two, matching the conditioning size . How does FiCS perform on more complex DAGs with larger maximum indegrees?
We thank the reviewer for this important question, which motivated us to test FiCS on graphs with maximum indegree four at . Increasing the maximum indegree makes FiCS statistically and computationally more demanding because the conditioning limit must be large enough to accommodate the true parent sets. Nevertheless, the effect was not prohibitive in the tested extension from maximum indegree two to four.
We conducted the experiment reported in Table 7 (Appendix G.1 of the revised manuscript) on ER graphs with an expected mean total degree of five (ER5) and scale-free (SF) graphs, both having a maximum indegree of up to four. FiCS with improved both F1 and SHD over in every setting and achieved the best results on both metrics among the baselines in seven of eight settings. These results extend the empirical support for FiCS to DAGs with greater maximum indegree and highlight the benefit of increasing to accommodate larger parent sets.
| Graph | FiCS (F1 / SHD) | FiCS (F1 / SHD) | FiCS (F1 / SHD) | Best baseline F1 | Best baseline SHD | |
|---|---|---|---|---|---|---|
| ER5 | 50 | .303 / 106.50 | .407 / 102.61 | .460 / 100.64 | .230 (NoGAM) | 151.29 (NoGAM) |
| Scale-free | 50 | .289 / 109.17 | .395 / 105.21 | .449 / 105.21 | .200 (NoGAM) | 176.69 (NoGAM) |
| ER5 | 100 | .443 / 84.07 | .558 / 74.17 | .617 / 68.43 | .459 (CaPS) | 87.83 (NoGAM) |
| Scale-free | 100 | .434 / 86.13 | .566 / 74.20 | .619 / 69.70 | .438 (CaPS) | 86.80 (NoGAM) |
| ER5 | 200 | .524 / 71.83 | .638 / 59.23 | .686 / 53.43 | .646 (CaPS) | 58.37 (SCORE) |
| Scale-free | 200 | .531 / 70.90 | .645 / 58.80 | .695 / 52.57 | .644 (SCORE) | 58.43 (SCORE) |
| ER5 | 500 | .575 / 64.93 | .672 / 53.37 | .722 / 46.90 | .746 (CaPS) | 45.60 (CaPS) |
| Scale-free | 500 | .576 / 64.90 | .687 / 51.63 | .744 / 43.70 | .732 (CaPS) | 48.23 (SCORE) |
At , all methods are compared on the 28 ER5 and 29 SF inputs for which CaPS returned a DAG. Each other row uses all 30 inputs.
Question 3. Linear versus fully nonlinear mechanisms
Appendix G.2 shows that FiCS peforms substantially better on linear mechanisms than on fully nonlinear causall discovery. Could the authors clarify the source of this sensitivity and its implications for nonlinear causal discovery?
Thank you for asking about this sensitivity. On the graphs from the mechanism-proportion experiment (Figure 5 in Appendix G.2 of the revised manuscript), all-GP mechanisms have a smaller first-step population Fisher gap and lower ordering accuracy than all-linear mechanisms. Giving FiCS a true topological order raises all-GP F1 from to . All-linear F1 with the true order is , so a difference remains at the parent-selection stage.
On each of the same 30 DAGs, we rescaled the linear coefficients to match the GP model's first-step population Fisher gap, the score difference between the best source and the best nonsource. We computed the linear gaps from the model covariances and estimated the GP gaps numerically. The mean gap was about in both settings. With the true order supplied, the rescaled linear and all-GP mechanisms had similar F1 ( and ). Their estimated-order F1 and ordering accuracy still differed. Rescaling also changes the parent scores. Theorem 3 (Section 5.1) applies to nonlinear mechanisms and relates recovery to the ordering and parent-omission margins, score estimation error, and the parent penalty.
| Mechanism | First-step population Fisher gap | Estimated order F1 | True order F1 | |
|---|---|---|---|---|
| All linear | .2374 | .995 | .940 | .938 |
| Rescaled all linear | .0588 | .777 | .535 | .665 |
| All GP | .0588 | .862 | .608 | .678 |
Means over the same 30 DAGs (). uses the estimated order.
Thanks for authors' detailed responses, which have addressed most of my concerns. Although some limitations remain, I will raise my score to 3.
We sincerely thank the reviewer for the careful reading of our responses and for raising the score. The reviewer's questions led us to evaluate FiCS in more complex settings, which strengthened the empirical support of the manuscript.
Response to Reviewer EHgh [1/2]
We appreciate the reviewer's questions on how FiCS behaves in more complex settings. They helped us test FiCS in higher dimensions and on graphs with maximum indegree four, and the question on fully nonlinear mechanisms led us to relate the lower accuracy on such data to a smaller ordering margin.
Weakness 1. Sensitivity to graph density and dimension
We thank the reviewer for recognizing the performance of FiCS in high-dimensional settings. We tested FiCS on denser ER5 and scale-free graphs with maximum indegree four at p=50.
With the matching cap q=4, FiCS led the four baselines on both F1 and SHD in seven of eight settings. The full comparison appears in our answer to Q2 and in Table 7 (Appendix G.1 of the revised manuscript).
Theorem 2 in Section 3.2 requires the cap q to be at least the maximum indegree for both graph families. At maximum indegree two, its mean F1 is similar on ER3 and scale-free graphs (.784 and .782 in Figure 4, Appendix G.1).
We explain the similar F1 across dimensions in our answer to Q1 and the sensitivity to nonlinear mechanisms in our answer to Q3.
Question 1. Accuracy as p grows
We appreciate that the reviewer noticed FiCS keeping its F1 even when p>n, which is the main benefit of its local scores. The reason is that each local score of FiCS uses at most q+1 variables regardless of p. More specifically, the score error in Theorem 4 (Section 5.2 of the revised manuscript) depends on the number of local queries only through L=q{log(ep/q)}+logp, which grows with the fixed cap q but only logarithmically in p. SCORE and NoGAM instead estimate density derivatives jointly over all p variables, and at p=200 and n=100 they attain mean F1 of .055 and .066, against .574 for FiCS (Figure 2 in Section 6.1).
Appendix F.4 reports two additional experiments that support this explanation. At n=100, the mean F1 of FiCS decreases only slowly as p grows to 1000.
Changing only the conditioning size for ordering from two to all predecessors lowers mean F1 at every p from 50 to 200, as shown in Figure 3 (Section 6.2). The loss grows with p because conditioning on all predecessors estimates each score over more variables, and at p=200 it lowered F1 on every one of the 30 graphs.