- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper extend the Causal Relevance Index (CRI)---which tests the quality of a causal discovery algorithm without access to the ground truth graph by aggregating p-values -- to times-series by introducing a conservative p-value. The approach is tested experimentally on the Causal Chambers wind-tunnel.
Critical strengths
- The paper adapt recent work to a harder setting for which few solutions exist in the absence of ground truth.
Critical weaknesses
(NB: I only have textbook knowledge, and read a few papers on the domain of causal discovery, so I am not an expert reviewer.)
I believe the paper contribution is poorly presented
Generally, the introduction of the objects are too handwaving
- The graph is never introduced formally, nor the problem at hand
- \mathcal{E} introduction would deserve a few sentences
- where the p-value come from should be discussed more
- the set to which the different objects belong are almost never given
- DAG acronyme used but not explained
- CRI is not explained enough
- causal stationarity is not defined
- Section 3.4 lacks an introduction. As a reader, it was hard to follow the author though process
- many domain specific concepts are mentioned once in the paper without being explained. The document is not self-contained for a general AI audience
articulation of the contribution not optimal
- no contribution paragraph
- the general problem should be better explained, as well as the specificity of time series: as someone who have not work on the topic, I which I better understood the failure modes of the other methods mentioned in the introduction
- connection with the literature should be improved.
- previous result should be re-explained and recontextualized in the context of the paper
Questions for authors
- I do not have specific questions whose answers could materially affect my assessment for this paper.
Clarification of the Problem, Definitions, and Contributions
We thank the reviewer for their careful reading and comments on the presentation. We acknowledge that several concepts deserve a more explicit introduction. Below, we clarify the problem and the organization of the results.
1. Problem and graph representation
We observe a multivariate time series Xt=(X1t,…,Xdt)∈Rd. The temporal causal graph contains one node for each variable at each time point. An edge Xit−τ→Xjt represents a direct causal influence from variable i to variable j at lag τ.
A DAG, or “directed acyclic graph,” denotes a directed graph containing no directed cycles. Our evaluation considers strictly positive lags, so edge orientation follows temporal order.
For a fixed maximum lag τmax, the candidate edge family comprises E={(i→j,τ):i,j∈{1,…,d}, i=j, 1≤τ≤τmax}. This set describes the candidate links, regardless of their inclusion in the estimated graph. The problem consists of evaluating the results of a discovery configuration (algorithm, CI test, parameters, and conditioning strategy) using the data and the tests performed, without access to the reference causal graph.
2. Origin of the p-values and definition of CRI
For each candidate edge e=(i→j,τ), a constraint-based procedure tests hypotheses of the form H0,e,S:Xit−τ⊥⊥Xjt∣S, where S denotes a set of conditioning variables. Each test returns a p-value pe(S)∈[0,1], calculated from a statistic and its distribution under the conditional independence hypothesis.
The finite family Setest contains the conditioning sets actually explored for this link. We define
\qquad \mathrm{CRI}_{\mathrm{TS}} =1-\frac{1}{|\mathcal{E}|}\sum_{e\in\mathcal{E}}p_e^{\max}.$$ The maximum reflects the search for a separating set: a high p-value for one conditioning set reduces the link's contribution to the criterion. Low p-values across all explored sets increase this contribution. CRI thus summarizes test results over a fixed candidate family, including candidates absent from the estimated graph. It depends on the discovery configuration and test outcomes, beyond the final graph alone. **3. Temporal specificity and the role of Section 3.4** Causal stationarity means that the structure of causal relationships remains invariant under time shifts: for a fixed lag, the presence of $X_i^{t-\tau}\to X_j^t$ does not depend on $t$. Strict stationarity instead concerns the invariance of joint distributions under time shifts. The statistical difficulty arises from dependence between successive observations. A test calibration assuming independent observations potentially underestimates the statistic's variance, resulting in excessive small p-values under the null hypothesis. Section 3.3 first establishes CRI properties under a p-value validity condition, R1. Section 3.4 then identifies conditions under which this validity extends to temporally dependent observations. Theorem 2 addresses Fisher-z, accounting for the variance induced by temporal dependence and establishing asymptotic guarantees. **4. Contributions and relationship to previous work** Strobl et al. (2019) construct bounds on edge-specific p-values from CI tests. Kunitomo-Jacquin et al. (2024) use these quantities to define CRI under an i.i.d. framework, notably for evaluating discretization strategies. Our contribution comprises three elements: formulation of the criterion over directed lagged links; guarantees based on marginal super-uniformity for at least one tested separating set per absent edge; and analysis of validity conditions under temporal dependence. Surrogate-based normalization additionally provides a null reference specific to each configuration. Experiments on synthetic data and the physical Causal Chambers apparatus then examine the relationship between this criterion, computed without ground truth, and metrics requiring a reference graph.