acceptodds
Withdrawn from ICLR 2027

Assessing Time-Series Causal Discovery without Ground Truth

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Assessing the relevance of a causal graph inferred from observational data remains limited when no reference graph is available, since standard structural and interventional metrics require ground-truth causal relations. We extend the Causal Relevance Index to lagged causal discovery by aggregating edge-specific p-value bounds over a fixed set of candidate lagged relations. The graph-level properties of the resulting index are shown to follow from a marginal condition on the conditional independence tests: for every absent edge, a super-uniform p-value obtained for at least one tested separating set provides a sufficient condition. This result requires no assumptions on the joint dependence structure among the edge-specific quantities. Conditions under which temporal dependence preserves this super-uniformity property are further derived, together with an empirical null normalization based on iterative amplitude-adjusted Fourier transform surrogates to account for finite-sample and test-specific effects. The resulting criteria are evaluated on nonlinear synthetic time series spanning several graph structures, sample sizes, causal discovery algorithms, and conditional independence tests, as well as on measurements from the Causal Chambers wind-tunnel system. Across these experiments, CRI follows the same trend as true-positive recovery without access to ground truth, while providing a continuous measure of the adequacy of the inferred graph to the conditional dependence structure supported by the data, as quantified by the edge-specific p-value bounds.

Primary area:
causal reasoning
Keywords:
causal discovery, time series, causal graph evaluation, conditional independence testing, causal relevance index, ground-truth-free evaluation, super-uniform p-values

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vZWzOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper extend the Causal Relevance Index (CRI)---which tests the quality of a causal discovery algorithm without access to the ground truth graph by aggregating p-values -- to times-series by introducing a conservative p-value. The approach is tested experimentally on the Causal Chambers wind-tunnel.

Critical strengths
  • The paper adapt recent work to a harder setting for which few solutions exist in the absence of ground truth.
Critical weaknesses

(NB: I only have textbook knowledge, and read a few papers on the domain of causal discovery, so I am not an expert reviewer.)

I believe the paper contribution is poorly presented

Generally, the introduction of the objects are too handwaving

  • The graph is never introduced formally, nor the problem at hand
  • \mathcal{E} introduction would deserve a few sentences
  • where the p-value come from should be discussed more
  • the set to which the different objects belong are almost never given
  • DAG acronyme used but not explained
  • CRI is not explained enough
  • causal stationarity is not defined
  • Section 3.4 lacks an introduction. As a reader, it was hard to follow the author though process
  • many domain specific concepts are mentioned once in the paper without being explained. The document is not self-contained for a general AI audience

articulation of the contribution not optimal

  • no contribution paragraph
  • the general problem should be better explained, as well as the specificity of time series: as someone who have not work on the topic, I which I better understood the failure modes of the other methods mentioned in the introduction
  • connection with the literature should be improved.
  • previous result should be re-explained and recontextualized in the context of the paper
Questions for authors
  • I do not have specific questions whose answers could materially affect my assessment for this paper.
1 comment
AuthorsOpenReview ↗

Clarification of the Problem, Definitions, and Contributions

We thank the reviewer for their careful reading and comments on the presentation. We acknowledge that several concepts deserve a more explicit introduction. Below, we clarify the problem and the organization of the results.

1. Problem and graph representation

We observe a multivariate time series Xt=(X1t,…,Xdt)∈RdX^t=(X_1^t,\ldots,X_d^t)\in\mathbb{R}^d. The temporal causal graph contains one node for each variable at each time point. An edge Xit−τ→XjtX_i^{t-\tau}\to X_j^t represents a direct causal influence from variable ii to variable jj at lag τ\tau.

A DAG, or “directed acyclic graph,” denotes a directed graph containing no directed cycles. Our evaluation considers strictly positive lags, so edge orientation follows temporal order.

For a fixed maximum lag τmax⁡\tau_{\max}, the candidate edge family comprises E={(i→j,τ):i,j∈{1,…,d}, i≠j, 1≤τ≤τmax⁡}.\mathcal{E}=\{(i\to j,\tau):i,j\in\{1,\ldots,d\},\ i\neq j,\ 1\leq\tau\leq\tau_{\max}\}. This set describes the candidate links, regardless of their inclusion in the estimated graph. The problem consists of evaluating the results of a discovery configuration (algorithm, CI test, parameters, and conditioning strategy) using the data and the tests performed, without access to the reference causal graph.

2. Origin of the p-values and definition of CRI

For each candidate edge e=(i→j,τ)e=(i\to j,\tau), a constraint-based procedure tests hypotheses of the form H0,e,S:Xit−τ⊥ ⁣ ⁣ ⁣⊥Xjt∣S,H_{0,e,S}:X_i^{t-\tau}\perp\!\!\!\perp X_j^t\mid S, where SS denotes a set of conditioning variables. Each test returns a p-value pe(S)∈[0,1]p_e(S)\in[0,1], calculated from a statistic and its distribution under the conditional independence hypothesis.

The finite family Setest\mathcal{S}_e^{\mathrm{test}} contains the conditioning sets actually explored for this link. We define

\qquad \mathrm{CRI}_{\mathrm{TS}} =1-\frac{1}{|\mathcal{E}|}\sum_{e\in\mathcal{E}}p_e^{\max}.$$ The maximum reflects the search for a separating set: a high p-value for one conditioning set reduces the link's contribution to the criterion. Low p-values across all explored sets increase this contribution. CRI thus summarizes test results over a fixed candidate family, including candidates absent from the estimated graph. It depends on the discovery configuration and test outcomes, beyond the final graph alone. **3. Temporal specificity and the role of Section 3.4** Causal stationarity means that the structure of causal relationships remains invariant under time shifts: for a fixed lag, the presence of $X_i^{t-\tau}\to X_j^t$ does not depend on $t$. Strict stationarity instead concerns the invariance of joint distributions under time shifts. The statistical difficulty arises from dependence between successive observations. A test calibration assuming independent observations potentially underestimates the statistic's variance, resulting in excessive small p-values under the null hypothesis. Section 3.3 first establishes CRI properties under a p-value validity condition, R1. Section 3.4 then identifies conditions under which this validity extends to temporally dependent observations. Theorem 2 addresses Fisher-z, accounting for the variance induced by temporal dependence and establishing asymptotic guarantees. **4. Contributions and relationship to previous work** Strobl et al. (2019) construct bounds on edge-specific p-values from CI tests. Kunitomo-Jacquin et al. (2024) use these quantities to define CRI under an i.i.d. framework, notably for evaluating discretization strategies. Our contribution comprises three elements: formulation of the criterion over directed lagged links; guarantees based on marginal super-uniformity for at least one tested separating set per absent edge; and analysis of validity conditions under temporal dependence. Surrogate-based normalization additionally provides a null reference specific to each configuration. Experiments on synthetic data and the physical Causal Chambers apparatus then examine the relationship between this criterion, computed without ground truth, and metrics requiring a reference graph.
Reviewer_Z31vOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies the Causal Relevance Index (CRI) to time-series data. For theory, they only require super-uniformity to prove some theoretical properties for CRI in the time-series case. They also investigate under which condition the p-value satisfies super-uniformity for time-series data. They also evaluate CRI on the results given by different time-series causal discovery methods.

Critical strengths

The problem this paper wants to address is one of the most critical problems in causal discovery. When we apply causal discovery to real data, in most cases we don't have ground truth. So it is worth discussing how we can assess the causal discovery results without ground truth.

Critical weaknesses
  1. This paper is an incremental paper of Causal Relevance Index (CRI). If a method is well-recognized and well-used in iid data, it is definitely valuable to extend it to time-series data. For example, PCMCI applies PC to time series data and addresses the statistical issue raised by applying the PC to non-iid data. However, unlike PC, the application cases or discussion of CRI is very limited after its introduction in 2024. So this submission should spend some time to argue applying CRI to time series data is important. For example, what is the advantage of CRI compared to other method for assessing time-series causal discovery like self-compatibility (Faller et al., 2024), permutation-based causal graph falsification (Eulig et al., 2025) and some score function like BIC used for causal discovery?

  2. For experimental results, CRI's performance is not impressive and lack of comparison to other methods. For example, in Table 2, PCMCI-ParCorr have the highest F1 but the CRI and CRI_norm is not the highest. More importantly, Table 1 and Table 2 is just comparing different PCMCI variants and PCMat-HSIC. I am more interested to see the comparison of CRI with other methods which can also assess causal discovery method without ground truth like self-compatibility.

Faller, Philipp M., et al. "Self-compatibility: Evaluating causal discovery without ground truth." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

Eulig, Elias, et al. "Toward falsifying causal graphs using a permutation-based test." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 25. 2025.

Schwarz, Gideon. "Estimating the dimension of a model." The annals of statistics (1978): 461-464.

Questions for authors

Score-based causal discovery methods have been well-used and well-discussed for many years. The score functions like BIC used in these methods also can assess how good the graph is to the data. What is the non-trivial difference between CRI and score functions when applying them to assess the graph from causal discovery methods?

Schwarz, Gideon. "Estimating the dimension of a model." The annals of statistics (1978): 461-464.

1 comment
AuthorsOpenReview ↗

CRI versus BIC: Temporal Extension and Interpretation of Results

We thank the reviewer for these comments. We clarify the distinction between CRI and model scores, the contribution of the temporal extension, and the interpretation of the experimental results.

1. Difference between CRI and BIC

BIC evaluates a graph together with a family of probabilistic models: its computation requires a likelihood, parameter estimation, and a parameter count. The same graph associated with different probabilistic families therefore potentially receives different scores. The resulting ranking reflects both graph structure and modeling assumptions.

CRI requires no such global probabilistic specification. Its computation uses the p-values from conditional independence (CI) tests performed on candidate links. For each link, it retains the largest p-value across the conditioning sets explored, then aggregates these quantities. It therefore depends on the selected tests, their validity, and the conditioning sets explored, without requiring a complete probabilistic model fitted to the graph.

This distinction directly benefits constraint-based procedures using nonparametric tests such as HSIC: CRI uses their test results, whereas evaluation through BIC would require an additional probabilistic modeling step.

The literature also distinguishes these objectives. Vowels et al. (2022, Section 3.5) separate edge-recovery metrics from model-fit criteria such as BIC. For time series, Howard and Kunze (2023) present BIC among discovery scores, then evaluate the graphs through precision, recall, and F1.

2. Temporal contribution and relation to other criteria

Self-compatibility and falsification address evaluation without ground truth under an i.i.d. framework. This assumption appears explicitly in Definition 2 of Faller et al. (2024) and Section 3.1 of Eulig et al. (2025). Their application to time series therefore requires specifying the representation of lagged links and the validity conditions for statistical procedures under temporal dependence.

Self-compatibility examines the compatibility of graphs learned from different subsets of variables. The falsification method compares graph-data inconsistencies against a baseline constructed through node permutations. A comparison in our temporal context would therefore require adapted versions, together with a justification of their applicability conditions.

Our contribution addresses precisely this transition of an evaluation criterion to temporal data. Dependence between observations changes the null distribution of CI statistics and, consequently, the validity of the p-values entering CRI. The extension therefore concerns the statistical foundations of the criterion beyond the introduction of lagged links.

The guarantees rely on an explicit marginal condition: for each absent link, the tested family contains at least one valid separating set whose test provides a super-uniform p-value. Under this condition, Pr⁡(pemax⁡≤α∣H0,e)≤α.\Pr(p_e^{\max}\leq\alpha\mid H_{0,e})\leq\alpha. The results include a bound on expected CRI under the global null and the bound in Proposition 1 on the number of true links among the candidates. These guarantees allow arbitrary dependence between the quantities associated with different links. Theorem 2 specifies conditions for temporal validity of Fisher-z. Surrogate-based normalization additionally provides a reference specific to each experimental configuration.

3. Origin of CRI and interpretation of Table 2

CRI builds on Strobl et al. (2019), who establish bounds on edge-specific p-values. Their analysis notably assumes a zero Type II error rate for CI tests: true adjacencies then remain throughout the search, allowing the tests required to compute these bounds. Building on this construction, Kunitomo-Jacquin et al. (2024) propose an aggregation intended to reproduce mean true-positive trends and examine this behavior empirically.

This origin clarifies the objective of CRI: summarizing the statistical evidence associated with candidate links. F1, in comparison, combines precision and recall. Our extension establishes aggregation guarantees under R1 without imposing a globally zero Type II error rate.

In Table 2, PCMCI-ParCorr achieves the highest F1 (0.333), with precision 0.275 and recall 0.423. PCMat-HSIC-MCI achieves precision 0.714, recall 0.192, and F1 0.303. Its normalized CRI also reaches the highest value (0.123). In this experiment, the maximum normalized CRI therefore coincides with the maximum precision, whereas the highest F1 corresponds to a procedure with higher recall.

This observation concerns the normalized criterion: raw CRI reaches its maximum for PCMCI+-GPDC. The table thus illustrates the behavior of both versions and their relationship with different edge-recovery metrics.

Reviewer_cw8uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends the Causal Relevance Index, originally proposed for selecting discretization strategies for i.i.d. data, to evaluate lagged causal discovery in time series without requiring a ground-truth graph. The proposed criterion is defined as one minus the average of edge-specific p-value bounds over a fixed set of candidate lagged edges, where each bound is the maximum p-value across the conditioning sets tested by a constraint-based algorithm. The paper provides theoretical results showing that the criterion satisfies several properties if at least one valid separating set is tested for each absent edge, without requiring assumptions on the dependence among p-values. It also derives conditions for the asymptotic validity of Fisher z-tests under temporal dependence. Furthermore, a surrogate-based normalization is proposed to account for different null levels across configurations.

Critical strengths
  1. The paper studies a very important and practical problem: how to evaluate time-series causal discovery without a ground-truth graph.
  2. The paper is well written and clearly structured.
  3. The theoretical results relax the zero Type II error assumption used in the earlier PC-p framework by requiring only one valid separating set with a super-uniform p-value to be tested for each absent edge. The results also do not require assumptions on the dependence among the edge-specific p-value bounds. The paper also considers the calibration of Fisher z-tests under temporal dependence and proposes a surrogate-based normalization to account for configuration-specific null levels.
  4. The experiments include both synthetic data and a real-world dataset with a physical reference graph.
Critical weaknesses
  1. As for the experimental section, the evaluation is not comprehensive. The positive association between normalized CRI and recall or F1 is mainly demonstrated for PCMat-HSIC across different sample sizes, which may be driven by the sample size rather than the quality of the criterion. Within each sample size in Table 1, the normalized CRI appears to favor precision rather than recall or F1. Additionally, the synthetic experiments are limited to small graphs with 4 to 7 variables and Gaussian innovations.
  2. The proposed CRI relies on CI-test p-values, which limits its applicability to constraint-based methods. It is unclear how the criterion can be applied to other types of causal discovery methods, such as score-based, functional, or Granger-type approaches.
  3. Since the edge-specific bound is defined as the maximum p-value over the conditioning sets tested by each algorithm, the CRI depends on the testing strategy. Algorithms that test fewer conditioning sets may obtain higher scores, which raises concerns about the fairness of comparisons across different algorithms.
  4. Regarding the theoretical section, the novelty is limited. The index follows the same definition as the original CRI, and Theorem 1 mainly follows from the linearity of expectation. Furthermore, the theoretical results mainly characterize the behavior of CRI under the null hypothesis, but do not establish whether a higher CRI indicates better graph recovery.
  5. On the wind-tunnel dataset, the graph with the highest CRI has a recall of only 0.19, which raises concerns about whether the criterion reflects overall graph quality.
Questions for authors
  1. Theorem 2 only establishes sufficient conditions for the validity of Fisher z-test p-values under temporal dependence. However, the experiments also use HSIC and GPDC, and the strongest results are obtained with PCMat-HSIC. How is condition R1 justified for these CI tests under temporal dependence?
  2. Condition R1 requires at least one valid separating set to be tested for each absent edge. However, PCMat-HSIC limits the maximum conditioning-set size to two, so it is unclear whether this condition is satisfied in the experiments. Can the authors clarify whether valid separating sets are always tested? It would also be helpful to report the empirical calibration of CI-test p-values for absent edges at the sample sizes considered.
  3. PCMat-HSIC is introduced without a citation. Is it a new algorithm proposed in this paper, and what is the motivation for including it in the evaluation?
2 comments
AuthorsOpenReview ↗

Theoretical justification of R1, separator coverage, and the role of PCMat-HSIC

We thank the reviewer for these questions. The following clarifications address test calibration, separator coverage, and the role of PCMat-HSIC.

1. Justification of R1 for HSIC and GPDC

Theorem 1 applies to CI tests whose p-values satisfy marginal super-uniformity under R1. Theorem 2 establishes sufficient conditions for Fisher-z. Its transfer argument extends to other statistics whenever their calibration provides asymptotically valid p-values.

For HSIC, Chwialkowski and Gretton (2014) establish shift-based calibration under stationarity and mixing conditions. For a fixed lag τ\tau, the test concerns the process of pairs (Xit−τ,Xjt)(X_i^{t-\tau},X_j^t). Their assumptions, applied to this process, provide the HSIC-specific conditions for asymptotic super-uniformity.

The additive model used in our simulations also supports conditional testing through residuals. When conditioning on exactly the target's parents, the residual computed with the exact regression function equals its innovation. This innovation has independence from the past and hence from the residual of the lagged source. The model therefore justifies residual independence for this separator.

For estimated regressions, a sufficient condition requires their estimation error to have a negligible effect at the scale of the null fluctuations. For HSIC, this condition takes the form

n[HSIC^n(R^X,R^Y)−HSIC^n(RX,RY)]→P0.n\left[\widehat{\mathrm{HSIC}}_n(\widehat R_X,\widehat R_Y)-\widehat{\mathrm{HSIC}}_n(R_X,R_Y)\right]\xrightarrow{\mathbb P}0.

The symbols R^X,R^Y\widehat R_X,\widehat R_Y denote estimated residuals, while RX,RYR_X,R_Y denote residuals computed with the exact regression functions. The normalized statistics then share the same limiting distribution. A calibration procedure consistently estimating this distribution preserves asymptotic validity. This condition specifies the required guarantee for the regressions and RFF approximation used in PCMat-HSIC.

GPDC admits the same transfer argument: the additive model justifies independence of the exact residuals, while valid calibration of the distance-correlation statistic and preservation of its null limit after regression estimation provide asymptotic super-uniformity.

Thus, Fisher-z provides the case developed in Theorem 2, HSIC has published temporal-calibration results, and the extension to estimated conditional procedures rests on the transfer conditions above. These guarantees concern the limit as sample size increases.

2. Conditioning-set size and separator coverage

R1 requires at least one valid separator to be tested for every absent edge. With conditioning sets limited to two variables, the relevant condition requires an admissible separator of size at most two and its inclusion in the tested family.

An explicit bound separates calibration and coverage. For an absent edge, let Ae,nA_{e,n} denote the event that a fixed valid separator Se∗S_e^* belongs to the tested family. Suppose

Pr⁡(pe,n(Se∗)≤u∣H0,e)≤u+ϵe,n,\Pr\left(p_{e,n}(S_e^*)\le u\mid H_{0,e}\right)\le u+\epsilon_{e,n},

and

Pr⁡(Ae,nc∣H0,e)≤δe,n.\Pr(A_{e,n}^{c}\mid H_{0,e})\le\delta_{e,n}.

Max-aggregation then gives

Pr⁡(pe,nmax⁡≤u∣H0,e)≤u+ϵe,n+δe,n.\Pr\left(p_{e,n}^{\max}\le u\mid H_{0,e}\right)\le u+\epsilon_{e,n}+\delta_{e,n}.

On Ae,nA_{e,n}, the maximum bounds pe,n(Se∗)p_{e,n}(S_e^*) from above. The argument accommodates dependence between separator selection and the p-values. Asymptotic super-uniformity follows when both errors converge to zero.

The bound distinguishes calibration error ϵe,n\epsilon_{e,n} from coverage error δe,n\delta_{e,n}. For the size-two search, the guarantee applies to configurations admitting such a separator, with inclusion probability tending to one. Autoregressive parents and admissible lags enter the definition of the conditioning family.

For the reported experiments, complete separator coverage retains the status of an assumption required for applying the theoretical results. Regarding empirical calibration, the reported tables contain graph-recovery metrics and CRI values; the justification provided here concerns asymptotic guarantees under the conditions above.

3. Status and motivation of PCMat-HSIC

PCMat-HSIC denotes our experimental implementation of a PC-style skeleton search over lagged links. It combines HSIC for marginal tests with HSIC applied to kernel-regression residuals for conditional tests.

The name identifies this combination within our study. The references document its methodological components. The paper's contribution concerns CRI and its properties; PCMat-HSIC serves as a discovery procedure in the evaluation.

Its inclusion allows us to examine CRI with a procedure different from the PCMCI variants, particularly its sensitivity to CI tests and conditioning sets.

Thank you for the clarifications. However, the response does not address the concerns raised in the weaknesses. I therefore keep my score.

Comments on the paper

1 comment
Area_Chair_mBbKOpenReview ↗

Please join the author-reviewer discussion

Dear the authors and reviewers,

We noticed the authors did not provide responses. Our author and reviewer discussion will end soon in 18 November (AOE). If you still have some issues to discuss, this may still be a good time to do it. We highly appreciate your time on discussing and exchanging thoughts to keep our ICLR standards. Many thanks!

Bests,
Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.