- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aims to make temporal causal discovery feasible for very large systems. It proposes VL-PCMCI, which combines tiled correlation screening, sparse storage, blockwise causal discovery, and cross-block refinement. The theoretical analysis identifies conditions for recovering the underlying graph and explains sources of false discoveries. Experiments examine computational scaling, graph recovery on synthetic systems, and applications to river discharge and financial time series.
Critical strengths
- The implementation addresses substantial computational and memory constraints in large-scale causal discovery. The experiments demonstrate that the proposed workflow can process systems far beyond the sizes considered by the reference implementation.
- The paper carefully distinguishes computational feasibility from statistical reliability. The analysis and diagnostic experiments explicitly identify failures caused by screening, missing conditioning variables, and repeated use of the same observations.
Critical weaknesses
-- The recovery guarantees do not directly support the pipeline used for the main scaling results. The recovery analysis requires retesting provisional decisions after assembling the necessary conditioning variables, together with calibration that accounts for selecting and testing candidates on the same observations. The main implementation leaves earlier within-block decisions unchanged and does not apply this calibration. Consequently, the theoretical safeguards are not part of the procedure whose large-scale accuracy and cost are reported. The independent-data confirmation experiment offers a possible remedy, but uses an additional trajectory at smaller scales. Evaluating a corrected procedure under a fixed total observation budget would clarify whether reliable recovery can retain the reported computational benefits.
-- The large-scale accuracy results depend on a particularly favorable signal structure. The main synthetic systems have sparse linear interactions and clearly detectable marginal signals. These conditions support the initial screening, but true parents can be removed when their effects cancel or are obscured by stronger indirect associations. Later refinement cannot recover an edge that never enters the candidate set. The paper demonstrates such failures in small diagnostic examples, which is useful, but these examples do not establish how performance changes as the difficulties become more common in a large system. Broader evaluation of heterogeneous interactions is needed to determine the range in which scaling preserves useful causal accuracy.
-- The reported parameter settings do not fully describe the behavior of the evaluated implementation. The implementation notes disclose that the selected significance level for parent selection was not passed to the batched test, which instead used a fixed default. The stated tuning therefore did not fully control the computation. The proposed method and the reference implementation also allocate their conditioning budgets differently, so matching the overall budget does not make their statistical decisions equivalent. Although the shared-hardware experiments are valuable, these discrepancies complicate the attribution of accuracy and runtime differences. The affected comparisons need to be repeated with corrected settings and justified conditioning choices.
-- The real-data evaluation provides limited independent support for practical causal discovery. The river benchmark reuses the same held-out windows across development iterations in which the scoring or conditioning rules changed, and the selected final configuration was not recorded. The authors appropriately describe the results as descriptive, but the reported advantage therefore remains difficult to reproduce and assess on unseen data. The financial application demonstrates operational feasibility without a reference graph, so it cannot resolve this uncertainty about causal accuracy. A reproducible evaluation on untouched observations would provide stronger support for transferring the synthetic results to real systems.
Questions for authors
Please see weaknesses.
3 comments
W3. PC level and conditioning budgets.
- The defect. The five batched CI calls in the PC step omitted the level, both within blocks and in the merging PC step, so every VL-PCMCI run removed parents at 0.05. MCI tests and reference PCMCI applied their levels as configured.
- Unaffected studies. The scaling studies configured 0.05, so they are unaffected.
- Affected studies. The same-hardware block-merging row and the river benchmark: both configured 0.01 and ran 0.05. The financial panel also configured 0.01 and most likely ran 0.05; we use it for scale only (App. A).
- In the paper. It now reports the applied level (0.05), with the configured value in parentheses. The defect is fixed.
- Reruns with the configured level applied:
- River development windows: mean F1 0.3438 against 0.3456 as run.
- Untouched river stations: 0.261 against 0.234 (inconclusive under our pre-declared rule).
- Same-hardware comparison: rerun as a 2×2, PC level as run vs as configured × each method's own conditioning rule vs one shared rule (the reference implementation's default). Five cells, three seeds each, one CPU thread and a 235-minute limit (App. C.7.1, Table 13).
- Configured level applied: block merging's F1 changes by at most 0.0007 in every cell, and its fit time falls to 0.39–0.96 of the as-run time. The completion frontier stays at 1,000 series (all three fits at 3,000 hit the limit). No conclusion changes.
- One shared conditioning rule: every gap between block merging and a baseline moves by at most 0.005 F1. Our rule fixed in advance counts a gap within ±0.005 as a tie. At 300 linear series, block merging's deficit to screened PCMCI goes from −0.0062 to −0.0049, from a loss to a tie. We therefore say that part of that gap comes from the conditioning rule, and we report both rows.
- Runtime: the shared rule makes block merging 2.2–15.5× faster and PCMCI 1.3–2.4× slower. One PCMCI fit at 300 series and all three screened-PCMCI fits at 1,000 hit the limit. Relative to the baselines, block merging's time under the shared rule is 0.04–0.37 of its value under each method's own rule, so the runtime gap widens under the shared rule.
- Still unequal: the BH families differ, per block for VL-PCMCI and global for the baselines.
- Reproduction check. Before any change was applied, the reruns reproduced all 60 original fits exactly. The unpatched code at 0.01 returns the same edges as the patched code at 0.05, which confirms 0.05 was in effect.
W4. River evaluation.
- The selection was recorded. Each method's selected configuration was saved with that evaluation. The submission's "not recorded" was wrong, and App. A and C.4 now list the selections.
- A one-shot evaluation on untouched observations. No calendar time is untouched, so we used stations no earlier evaluation touched: 9 panels of 9–22 stations, 18 held-out windows, every method frozen, a test confirming no station-time overlap, and decision rules fixed in advance.
- Results:
- Against PCMCI: tuned VL-PCMCI 0.234, tuned PCMCI 0.236 (difference −0.001, 95% interval [−0.034, 0.031]). The earlier advantage did not replicate.
- Against sparse VAR: sparse VAR was more accurate (0.322; −0.088, [−0.130, −0.038]).
- Against its own default: the tuned configuration fell below VL-PCMCI's default (0.304), which indicates that tuning overfit the reused windows, as you suspected.
- Cost: VL-PCMCI remained 2.8× faster than PCMCI.
- With every method at its default (post hoc, not a pre-declared contrast): VL-PCMCI is ahead of PCMCI on the untouched stations (+0.054, [0.022, 0.087]) and level with it on the 30 earlier windows (−0.011, [−0.031, 0.007]); sparse VAR is ahead on both. So neither configuration leads PCMCI on both sets. These data support equal accuracy at lower cost against PCMCI (2.8× to 26× lower fit time), and no accuracy advantage.
- In the paper. §5.6 and App. C.4 now report this as stated.
- The final pass on these windows. Applied afterwards to the held-out windows (post hoc, a second use of them), the final pass leaves the default policy's F1 unchanged (0.304 to 0.305) and lowers the tuned policy's by 0.018. No contrast with a baseline changes sign. Two that had included zero now exclude it, both against VL-PCMCI, and one of four predictions failed. The selective level is not valid on these autocorrelated rows, so this describes what the pass does there, not error control. The rules that count all lagged pairs (a third use of the windows) do not change the result either (App. C.15.4).
- The financial graph. As you say, it has no ground truth. The paper uses it to show operational scale only, and §7 says so.
- Caveats. The panels are smaller than the original 32–128, and the stations share years and weather with those used before.
Dear Reviewer 6qam,
A brief follow-up, since the discussion period closes soon. Your first two concerns were the gap between the guarantees and the evaluated pipeline (W1) and the favourable signal structure of the main systems (W2). The revision evaluates the complete procedure at the same observation budget (FDP 0.17–0.26%, recall 0.9998 from 10⁵ to 2×10⁶ variables; Table 2, §5.3), and adds a difficulty sweep whose screening losses we predicted before any run and report in full (App. C.11).
If these do not address your concerns, or if anything in our replies is unclear, we would be glad to answer in the remaining hours. If they do, we would be grateful if you could update your review. Thank you for your time.
Thank you for a careful review. Each of your four points led to a new evaluation with its predictions fixed in advance.
W1. Guarantees vs. the evaluated pipeline; fixed observation budget.
W2. Favourable signal structure.