acceptodds
Under review as a conference paper at ICLR 2027

Uncertainty-aware Causal Decision Making via Effect Bound Decomposition

15 views

Abstract

Causal inference from observational data can provide strong evidence for finding the best action in a decision-making scenario without having to perform expensive randomized trials. The causal effect of an action is often not pointwise identifiable even with infinite data due to unobserved confounding factors. Furthermore, having only finitely many samples adds another layer of uncertainty to causal effect estimation. Several existing methods can be used to obtain upper and lower bounds to the causal effect, ranging from symbolic methods to the more recent neural network-based approaches, which implicitly incorporate both sources of uncertainty. However, these methods do not inform whether collecting more samples may or may not help identify the best action from observational data, leaving experts in the dark about their data collection strategies. We address this problem with a novel framework that can distinguish the range of causal effect values that might be eliminated by collecting more samples from the range of values that, with high probability, cannot be eliminated with more observational samples. We show that this partitioning can be obtained by solving max-min and min-max optimization problems. We leverage neural causal models to approximately recover this decomposition in practice. We demonstrate via experiments on synthetic and real-world datasets that our algorithm can determine when collecting more samples will not help determine the best action. Our framework can help practitioners decide when to resort to non-observational studies or seek to measure some of the unmeasured confounders for optimal decision-making.

Primary area:
causal reasoning
Keywords:
causal inference, causal graphs, neural causal models, causal effect estimation, sample uncertainty, non-identifiable uncertainty

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4jQ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops causal decision making framework under partial identification. Finite sample uncertainy is separated from non-identification irreducible (modelling) uncertainty by taking intersections and unions of partially identified effect sets over a confidence set of observational distributions. This yields a rule for deciding whether to act, collect more samples, or observe additional variables. The resulting nested optimization is approximated using deep causal models (UA-DCM) and evaluated on synthetic and real-world data.

Critical strengths

Problem formulation and presentation: I appreciate the clear exposition throughout the paper. Paper is clearly written and easy to follow Useful connection between partial identification and data collection. The framework turns identification bounds into actionable statements which seems clearly useful to me.

Critical weaknesses
  1. Limited methodological novelty / adaptive decision making: The core construction is a direct application of finite-sample uncertainty quantification on top of partial identification: construct a confidence set over observational distributions, obtain an identified set for each, and take their intersection/union. The more interesting opportunity and a natural extension is adaptive data acquisition, but this is not developed. "OBSERVE" diagnoses that additional variables are required without deciding which variables should be acquired. I would have liked this developed further to deepen the novelty and contribution.

  2. Theory-to-algorithm gap and insufficient validation: The guarantees apply to exact optimization over the confidence set and exact causal-effect bounds, while UA-DCM approximates both using finitely sampled candidate distributions and non-convex neural optimization. The experiments do not establish how accurately UA-DCM recovers the ideal decomposition or its resulting decisions. In the synthetic settings where closed-form bounds are available, the paper can and should separate the ideal decomposition, confidence-set approximation, and DCM approximation and evaluate endpoint and RETURN/COLLECT/OBSERVE decision errors across repeated datasets. Looking at the experiments the approximation seems to sometimes struggle quite a bit compared to baselines (fig 4) - I would want to understand if the issue is the complexity of implementation or possibly in the theory.

Questions for authors
  1. In the synthetic settings with closed-form causal bounds, how closely does UA-DCM recover the exact inner/outer decomposition? In particular, across repeated datasets, how often does it produce the same RETURN/COLLECT/OBSERVE decision?

  2. Can the framework be extended to choose adaptively which additional variable or information source to acquire, rather than only concluding that additional variables should be observed?

6 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2

We thank the reviewer for the constructive review. We highly appreciate that the reviewer found the paper clearly written and easy to follow, the connection between partial identification and data collection useful, and the actionable statements clearly useful. Below, we address their concerns.


W1. Novelty.

"Limited methodological novelty... a direct application of finite-sample uncertainty quantification on top of partial identification"

Our main contribution is a finite sample decision framework that separates sampling uncertainty from identification uncertainty and connects it to the decision-making problem. We agree that the construction is conceptually simple. However, it is important and novel for decision-making problems, and has not been studied before. Moreover, obtaining and exploring informative confidence regions for the causal bounds is not trivial in general.


W1 / Q2. Adaptive data acquisition.

"Can the framework be extended to choose adaptively which additional variable or information source to acquire ...?"

We agree with the reviewer that adaptive data acquisition is an interesting next step. Selecting which variables to acquire raises several challenges:

  1. How much does a variable narrow the bound compared to other choices?
  2. How do unobserved confounders and the causal structure affect this choice?
  3. What role do acquisition costs play? The causal structure may suggest a variable that is costly to acquire or rarely observed in practice.

Formally characterizing these challenges and addressing them is a substantial separate research problem, which we leave to future work.

Nonetheless, a few directions are:

  • Instruments: in extensive experiments (Figs. 4, 6b and 6c, Table 3), we show empirically that instrumental variables add constraints, so the bounds become typically tighter and never wider than the bounds without them; this is why our Observe step suggests instruments (Sec. 4.3). For example, in our two-IV SCM, IP⋆I_{P^\star} shrinks from [−0.34,0.66][-0.34, 0.66] with no instrument to [−0.20,0.57][-0.20, 0.57] with one and [0.06,0.40][0.06, 0.40] with two. On the Parents' Labor Supply data, adding the same-sex instrument shrinks the non-ID region from [0.20,0.87][0.20, 0.87] to [0.32,0.77][0.32, 0.77] (Fig. 6b).
  • Confounders: observing all unobserved confounders between XX and YY makes the effect identifiable (e.g., by backdoor adjustment), which removes the non-ID region entirely.
  • Graph-based screening: since the ADMG is known, the query is non-identifiable exactly when the graph contains a hedge for it [1]. Running the ID algorithm on the graph with a candidate variable treated as observed shows whether observing it removes the hedge and makes the query identifiable; any such variable is a direct candidate to acquire.

Fig. 4. Implementation or theory.

"the approximation seems to sometimes struggle quite a bit compared to baselines (fig 4) - I would want to understand if the issue is the complexity of implementation or possibly in the theory."

The gap in Fig. 4 is an approximation error in the implementation, not a problem with the theory, and it is fixed by simple hyperparameter tuning. The DCM estimates the data fit and the ATE with Monte Carlo latent draws. The original results used 256 draws per step; we re-ran with 4096 draws per step, and the DCM now closely matches the closed form (the same decision in 98% of runs at M=300M=300). Moreover, prior work shows theoretically that bounds obtained by optimizing over sufficiently expressive neural causal models recover the true bounds [2, 3].

References

[1] Identification of joint interventional distributions in recursive semi-Markovian causal models.

[2] Partial identification of treatment effects with implicit generative models.

[3] The causal-neural connection: Expressiveness, learnability, and inference.

AuthorsOpenReview ↗

Rebuttal Part 2/2

W2 / Q1. Separating the error sources.

"... should separate the ideal decomposition, confidence-set approximation, and DCM approximation and evaluate endpoint...across repeated datasets."

To address the reviewer's concern, we use the closed-form bound as a baseline and compare it with UA-DCM. This separates the ideal decomposition and the confidence-set approximation from the DCM approximation. Below we discuss three sources of error.

  1. Bound error. For the bow and IV graphs, the Balke-Pearl bounds are exact and tight, so each candidate's IPI_P is computed without error. The DCM, in contrast, approximates the bound for any graph and distribution; it is trained to match the observed distribution only approximately (within a small total variation distance), so it can reach slightly more extreme effects than an exact fit, and its bounds are about 0.015 to 0.02 wider per end.
  2. Confidence-set coverage. We use the Hoeffding-Bonferroni set, which contains P⋆P^\star with probability at least 1−α1-\alpha and is conservative in practice. At M=300M=300, in every setting, the outer region contained IP⋆I_{P^\star} and the inner region stayed inside IP⋆I_{P^\star} in 100% of datasets, so it is not a source of error.
  3. Number of candidates MM. We extended the Fig. 4 experiment to M=10M = 10 and 300 candidates explored in the ϵ\epsilon-ball. With small MM, the inner region is wider than the correct inner region and the outer region is narrower than the correct outer region. Below we show the evaluation.

Endpoints across repeated datasets. We compare the closed form (CF) and the DCM on the same datasets and candidates (both instruments, α=0.05\alpha=0.05, 20 replicates). At fixed nn, the DCM closely matches CF's endpoints and takes the closed form's decision in 98% of runs at M=300M=300 (87% at M=10M=10).

NOuter, CFOuter, DCMInner, CFInner, DCM
M=300M=300
1k[-0.192, 0.591][-0.199, 0.599]EmptyEmpty
3k[-0.102, 0.503][-0.117, 0.516][0.198, 0.235][0.196, 0.238]
10k[-0.025, 0.463][-0.046, 0.481][0.149, 0.316][0.140, 0.333]
M=10M=10
1k[-0.094, 0.502][-0.099, 0.508][0.175, 0.254][0.173, 0.254]
3k[-0.055, 0.439][-0.066, 0.452][0.128, 0.291][0.120, 0.296]
10k[+0.005, 0.431][-0.014, 0.449][0.105, 0.344][0.088, 0.362]

"Empty" means maxmin =a= a exceeds minmax =b= b.


"and RETURN/COLLECT/OBSERVE decision errors"

Decision errors across repeated datasets. We re-ran the Fig. 4 scenario sequentially (α=0.05\alpha=0.05). The analyst starts with (X,Y)(X, Y) only. Observe adds the next instrument (I1I_1, then I2I_2) for the same units; Collect increases nn. Each cell reads action (N = sample size). For both methods, every Return has the correct sign and every Observe occurs where 0∈IP⋆0 \in I_{P^\star}.

Evaluation 1. Ai is the ii-th action (α=0.05\alpha=0.05; 20 replicates; % is the share of runs following that trajectory). The following table shows the most frequent trajectory across 20 independent replicates.

MMethod%A1A2A3A4A5A6A7A8A9A10
300CF65%Observe I1 (N=30)Collect (N=30)Collect (N=100)Collect (N=300)Observe I2 (N=1k)Collect (N=1k)Collect (N=3k)Collect (N=10k)Return (N=30k)
300DCM50%Observe I1 (N=30)Collect (N=30)Collect (N=100)Collect (N=300)Observe I2 (N=1k)Collect (N=1k)Collect (N=3k)Collect (N=10k)Collect (N=30k)Return (N=39k–51k)

Evaluation 2. For each step, the table below gives the % of runs whose action at that step matches the reference path (the most common closed-form path at M=300M=300):

A1A2A3A4A5A6A7A8A9A10
ReferenceObserve I1 (N=30)Collect (N=30)Collect (N=100)Collect (N=300)Observe I2 (N=1k)Collect (N=1k)Collect (N=3k)Collect (N=10k)Return (N=30k)ended
MMethodA1A2A3A4A5A6A7A8A9A10
300CF100100100957075100959095
300DCM8080100955560951001010
10CF1009590404090803030100
10DCM1001008535308585553580

The DCM leaves the reference path mainly at A9: its bounds are slightly wider (approximate fit), so it returns one step later at N= 39k–51k. Every run of both methods returns with a correct action. We will add the full experimental setup with further details (e.g., M=100M=100) to the paper.

We are happy to address any further concerns. If we have addressed all, we kindly request the reviewer to consider increasing the score.

Thank you for the detailed response and additional experiments. I appreciate the effort, and the new results substantially improve the empirical validation of the proposed approach.

Nevertheless, my overall assessment of the contribution remains largely unchanged. While I find the decision-oriented framing practically useful, the core methodology remains a relatively straightforward application of established ideas from partial identification and finite-sample uncertainty quantification. In my view, the paper offers limited additional methodological or conceptual insight, and I do not find the overall research contribution sufficiently substantial (for ICLR).

I therefore maintain my score of 2 (Weak Rejection).

Reviewer_gCZNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Based on SCMs/ADMGs and deep causal models, the paper distinguishes the finite-sample range of causal effects into structurally unidentifiable components and components that may shrink as the sample size increases. Based on this distinction, it recommends one of three actions: making a decision directly, collecting additional samples, or measuring new variables.

Critical strengths
  1. The paper investigates the question of whether to continue sampling or measure new variables, which has significant practical value for decision-making.

  2. The paper combines theoretical partial identification analysis with deep causal models to address causal graphs for which analytical bounds are difficult to derive. Beyond providing intervals for causal effects, the method also aims to recommend three types of follow-up actions: Return, Collect, and Observe.

Critical weaknesses
  1. The paper does not analyze how the discretization error introduced by the ε\varepsilon - net and the optimization error of the neural network affect the final identified intervals and the validity of the 1−α1-\alpha coverage guarantee.

  2. The experiments do not validate the interval coverage, decision error rate, or reliability of the Observe or Collect recommendations through repeated sampling.

  3. The paper lacks key ablation studies on important parameters, including the size of the ε\varepsilon-net, the number of candidate distributions, the number of training epochs, and the confidence level.

  4. The paper contains numerous grammatical errors, redundant citations, inconsistent notation, and figures and tables with excessively small fonts, which substantially hinder readability. For example, L34–36 “which... more?”, L45 “want ot”, and L457 “contains contains”. The manuscript would benefit from careful proofreading and substantial improvements in writing and formatting in a subsequent revision.

Questions for authors

Please refer to weaknesses.

7 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2

We thank the reviewer for the careful review and constructive feedback. We appreciate that the reviewer found our core research question of significant practical value, and recognized our combination of partial identification with deep causal models for decision making.


W1 to W3. Error analysis and ablations.

W1: "... how the discretization error introduced by the ϵ\epsilon-net and the optimization error of the neural network affect ... the validity of the coverage guarantee."

We address these concerns with our existing theoretical and empirical analysis, together with additional experiments.

With finite data, the true distribution P⋆P^\star is unknown; we only know a confidence set CC of plausible distributions that contains P⋆P^\star with probability at least 1−α1-\alpha. For each P∈CP \in C, we compute the range of causal effects IPI_P. The outer region (the union over CC) contains every plausible effect value, and the inner region (the intersection over CC) is the part that remains uncertain even with infinite data. In practice, we evaluate only MM candidates from CC (the ϵ\epsilon-net) and compute each range with a neural causal model (DCM); these are the two approximations the reviewer asks about. The new experiments use SCM 1 (two instruments, Fig. 4; α=0.05\alpha=0.05; true range IP⋆=[0.060,0.396]I_{P^\star}=[0.060, 0.396]) and compare against closed-form (CF) bounds: the analytical Balke-Pearl bounds for this IV graph, which are tight, i.e., they equal the true range IPI_P of each candidate.

1. ϵ\epsilon-net error and the number of candidate distributions MM.

With finitely many candidates, the most extreme distributions in CC may not be evaluated. Omitting them can only make the outer region narrower and the inner region wider than their values over the entire set CC. App. D.3 (Lemma 4, Corollary 3) quantifies this error for the binary bow-backdoor graph: if every distribution in CC is within total variation distance ϵ\epsilon of an ϵ\epsilon-net point, the estimated minmax and maxmin of P(y∣do(x))P(y\mid do(x)) are within 3ϵ3\epsilon of their true values, so a finer ϵ\epsilon-net yields a smaller error. The radius of the ϵ\epsilon-net is the confidence radius (determined by α\alpha and nn; App. E.2).

Experiment. We compute CF bounds for every candidate and vary only MM. Mean endpoints over 100 repeated datasets:

nOuterInner
M=300M=300
1k[-0.188, 0.585]empty (0.287 > 0.148)
3k[-0.094, 0.517][0.212, 0.248]
10k[-0.024, 0.468][0.148, 0.321]
M=100M=100
1k[-0.169, 0.565]empty (0.256 > 0.170)
3k[-0.080, 0.504][0.194, 0.262]
10k[-0.015, 0.461][0.141, 0.327]
M=10M=10
1k[-0.091, 0.496][0.178, 0.244]
3k[-0.036, 0.464][0.138, 0.304]
10k[+0.007, 0.438][0.113, 0.348]

Observation: A finer ϵ\epsilon-net keeps the estimated regions valid, and more samples make them tight around IP⋆I_{P^\star}:

  • Sample size. The outer region shrinks and the inner region grows as nn increases.
  • Number of candidates. The regions computed over the entire set CC always enclose IP⋆I_{P^\star} from both sides (outer ⊇IP⋆⊇\supseteq I_{P^\star} \supseteq inner), and more candidates bring our estimates closer to these regions; for M≥100M \ge 100, this held in every run. With M=10M=10, the regions appear closer to IP⋆I_{P^\star} (e.g., outer [+0.007,0.438][+0.007, 0.438] at 10k) only because extreme distributions are omitted; as a result, the outer region occasionally excludes part of IP⋆I_{P^\star} and the inner region occasionally extends beyond it (item 2).

2. Neural optimization error and coverage.

Coverage means that the outer region contains the true range IP⋆I_{P^\star} and the inner region lies inside it. Both hold whenever P⋆∈CP^\star \in C, which occurs with probability at least 1−α1-\alpha (Proposition 1; App. E.3). With an exact fit and exact optimization, neural causal models recover the true ranges for discrete variables [1]. Our DCM matches the observed distribution only approximately (within a small total variation distance), so it admits slightly more models, which makes its ranges slightly wider and therefore more conservative.

Experiment. On 20 datasets of SCM 1, we trained the DCM and compared it with CF on exactly the same datasets and candidates (nn=10k; both columns use the same 20 datasets):

MMOuter, CFOuter, DCMOuter contains IP⋆I_{P^\star} (CF / DCM)
300[-0.025, 0.463][-0.046, 0.481]100% / 100%
100[-0.016, 0.455][-0.037, 0.473]100% / 100%
10[+0.005, 0.431][-0.014, 0.449]95% / 100%

Observation: The DCM bounds are about 0.01 to 0.02 wider at each end, so the DCM is slightly more conservative: its outer region contained IP⋆I_{P^\star} in every run, and its inner region lay inside IP⋆I_{P^\star} for M≥100M \ge 100.

AuthorsOpenReview ↗

Rebuttal Part 2/2

3. Decision errors (Return, Collect, Observe).

W2: "... do not validate the interval coverage, decision error rate, or reliability of the Observe or Collect recommendations ..."

W3: "... the size of the ϵ\epsilon-net, the number of candidate distributions, the number of training epochs, and the confidence level."

The method returns an action when the entire outer region lies on one side of 0, i.e., every plausible model agrees on the best action (Theorem 4). It recommends observing new variables when the inner region contains 0, i.e., the decision remains ambiguous even with infinite data, with probability at least 1−α1-\alpha (Proposition 1(ii)). Otherwise, it recommends collecting more samples. A wrong decision therefore requires one of the errors analyzed in items 1 and 2.

Experiment. We performed additional experiments on SCM 1 (Fig. 4), running its scenario sequentially over 20 repeated datasets with both CF bounds and the DCM, at M=300M=300 and M=10M=10. Starting from (X,Y)(X, Y), Observe adds I1I_1 and then I2I_2, and Collect increases nn.

Observation: We summarize the observations here; please check the full tables in our response to Reviewer 4jQ3.

  • No decision errors: For both methods, every Observe occurred where 0∈IP⋆0 \in I_{P^\star}, and every Return selected the correct action do(X=1)do(X{=}1).
  • Typical path: The most common path (65% of runs with CF bounds) observes I1I_1 at nn=30, collects up to nn=1k, observes I2I_2, and then collects until it returns do(X=1)do(X{=}1) at nn=30k. The DCM follows the same path (50% of runs) but, owing to its slightly wider bounds, returns one step later, at nn=39k to 51k.
  • Agreement: At a fixed sample size, the DCM makes the same decision (Return or Collect) as the CF bounds in 98% of runs at M=300M=300 and 87% at M=10M=10; every disagreement is the DCM collecting where the CF bounds return.
  • Number of candidates: With M=10M=10, the outer region is narrower (item 1), so the CF bounds return more often, while the slightly wider DCM bounds more often lead to Collect; agreement therefore drops to 87%. A finer exploration of the ϵ\epsilon-net (M=300M=300) yields more stable decisions.

4. Confidence level.

A higher confidence level yields a larger set of plausible distributions, so the outer region widens and the inner region shrinks: the method becomes more conservative and requires more data to decide (App. E.2).

Experiment. App. F.2 varies the confidence level on SCM 1 with 10k samples: the outer region is [0.02,0.45][0.02, 0.45] at 70%, [0.02,0.48][0.02, 0.48] at 90% and [−0.02,0.48][-0.02, 0.48] at 95%, so the decision changes from Return to Collect. Over 100 datasets, α=0.10\alpha=0.10 versus 0.050.05 yields Return rates of 6% versus 4% (SCM 1, 10k), with coverage unchanged.

5. Sample size, graph size and misspecified graphs.

More samples shrink the set of plausible distributions: the outer region shrinks at rate O(1/n)O(1/\sqrt{n}) (Proposition 2, App. D), and the inner region grows toward IP⋆I_{P^\star} (Eq. 6), becoming non-empty once n>N⋆n > N^\star (Theorem 7). All guarantees assume that the causal graph is correctly specified (Assumption 3, App. A).

Experiment. App. F.2 (3k, 8k and 16k samples) and Fig. 6b show the outer region shrinking with nn; Fig. 6a shows correct decisions with 8, 12 and 18 variables; App. F.6 shows that misspecified graphs loosen the bounds but still enclose the correct ones.

W4. Writing and formatting.

"... numerous grammatical errors, redundant citations, inconsistent notation, and figures and tables with excessively small fonts ..."

We apologize for these errors. We have performed a detailed check of the full manuscript, fixed all such issues, and uploaded a revised draft.

We look forward to further discussion. If we have addressed your concerns, we kindly request that you consider increasing your score.

References

[1] The causal-neural connection: Expressiveness, learnability, and inference.

Dear Reviewer gCZN,

Thank you very much for your time and effort in reviewing this submission. The authors have now submitted their rebuttal, and the discussion period is nearing its end. We would greatly appreciate it if you could take some time to carefully review the authors’ responses and share your assessment of whether your concerns have been adequately addressed. Thank you again for your valuable time and contributions to the review process.

Reviewer_WWJROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper attempts to address a long-standing question in causal inference, that is how to distinguish between uncertainty due to non-identifiability and uncertainty due to lack of sufficient samples from observational distribution. It develops a theoretical framework to define and estimate those uncertainty regions, which it calls nonID uncertainty and sample uncertainty, respectively. In particular, it first starts by proving that the set of causal effects compatible with a feasible uncertainty ball around the empirical observed distribution form a contiguous interval (both in identifiable and non-identifiable settings) and concludes that it is enough to know the lower and upper bounds of the effect intervals. The paper then defines four min-min, min-max, max-min, max-max quantities over causal effects, where the first optimization is over the set of feasible observed probabilities, and the second one is over the set of SCMs that are compatible with a observed probability candidate. It then proves that the nonID uncertainty and sampling uncertainty regions could be derived from such quantities. Finally, the authors provide an algorithm to estimate those four quantities using minimax optimization, relying on neural causal models [1] to manifest the underlying SCM in a differentiable way and a Lagrangian optimization. Specifically, to search over the space of probability distributions, they follow a heuristic approach to identify a finite candidate set and loop over them. The goal of this algorithm is to first decide what the non-identifiable set is, and then use it to decide whether making a decision is possible, or whether more observational data is needed, or that the set of observed variables need to be expanded. Finally, a set of simulated and real-world experiments to show how the proposed algorithm performs with access to more samples or observed variables and how it adapts to large graphs.

--References--

[1] Xia, Kevin, et al. "The causal-neural connection: Expressiveness, learnability, and inference." Advances in Neural Information Processing Systems 34 (2021): 10823-10836.

Critical strengths

(i) First, the problem discussed in the paper is highly practical and well-motivated. Despite all the bitter-lesson arguments about just scaling compute and data in AI, causality tells us about the existence of settings that could not be resolved by just adding more data. This paper studies such settings concretely and attempts to distinguish the nonID uncertainty from sampling uncertainty. (ii) I also enjoy the decision-oriented framing. In many cases, the goal of effect estimation is not just getting to know the effect alone, but taking an appropriate action accordingly. The paper introduces a return/collect/observe framing that's intuitive and useful. (iii) The developed theoretical framework is agnostic to the bound estimators (whether they come from solving an LP or using an NCM), even though an estimation algorithm is provided.

Critical weaknesses

I have a few concerns about (i) what's actually being proved theoretically and what the paper claims and (ii) the theoretical results/proof precisions. I'll go into details one by one:

--Overclaiming the theoretical contribution--

The paper, in some places, claims/defines the nonID region as the uncertainty that will not be removed by sampling more data, while the sampling uncertainty as the reverse idea (e.g., "... that will not be removed with high probability by increasing the number of samples" in Introduction or "... can distinguish the range of causal effect values that might be eliminated by collecting more samples from the range of values that ... cannot be eliminated" in the abstract). However, these claims are only partly correct. Specifically, what Proposition 1 proves is that the defined nonID uncertainty region only is a subset of the true non-identifiable region and not the actual non-identifiable region itself. In other words, it is just an inner approximation, that can grow as we observe more observed data [the paper itself say so, e.g., in Section 4.2: "typically widens toward IP⋆\mathcal{I}_{P^\star}"]. Equivalently, the defined sampling uncertainty region includes values that are non-identifiable. In my view, the honest framing here should be that the paper provides a one-sided certificate about non-identifiability rather than saying it genuinely provides a decomposition between the sampling/identifiability uncertainties [e.g., in the conclusion section "novel decomposition of epistemic uncertainty"]. A simple suggestion is to present IP⋆\mathcal{I}_{P^\star} as the true nonID region and the current definition (the intersections) as an inner approximation of that.

A related concern is about the reliance of the results on ∩IP\cap \mathcal{I}_{P} being non-empty (e.g., in Theorem 3). It seems that there are few reasonable scenarios that this set will be empty. For example, if the effect is identifiable and the confidence set C\mathcal{C} has more than one element, then the intersection of the intervals will be empty since the intervals are simply points. Another example is when the effect is non-identifiable, but the identified set is narrow relative to the confidence set (e.g., in strong instrumental variable settings) as Theorem 7 already hints on. I believe having an honest and comprehensive study of such settings would strengthen the paper.

--Imprecise Theory--

A good part of the paper is theory. However, there are several cases where the terminology is not well-defined or proofs are not sound as written. I provide some examples:

  • Definition 3 is imprecise. It is not specified what "sufficiently high-dimensional" actually mean, or how to design/test for an "expressive enough" neural network.

  • Section 2 refers to Theorem 8 (in appendix) for details on how non-identifiable effects are bounded, but the theorem statement is only about identifiable interventional quantities. Moreover, the paper does not show that its choice of NCMs in the experiments can represent every SCM compatible with the given graph.

  • Theorem 2 relies on "conditions of Lemma 3" and a "finite-state causal graph" but neither is discussed in the main text. Moreover, the paper relies on "the set P\mathcal{P} is connected" without defining what that means.

  • Proposition 2 is not proved.

Questions for authors
  1. In what sense the intersection ⋂P∈CIP\bigcap _{P \in \mathcal{C}} \mathcal{I}_P is the nonID uncertainty, given that it changes with the sample size? Is there a result showing that the intersection converges to IP⋆\mathcal{I}_{P^\star} as we get more data?

  2. Why/when should the intersection of the effect intervals be non-empty? What should the practitioner do when it is empty? Is it possible to distinguish an empty intersection due to identifiability from one due to a narrow non-identified set?

  3. What is the definition of a "connected set" of observational distributions and of a "finite-state" causal graph? moreover, where is Lemma 3 used in the proof of Theorem 2?

  4. Could the authors provide a proof of Proposition 2?

  5. If we are interested in statements like "with probability at least 1−δ1-\delta", how can one choose the radius of the confidence set ϵ\epsilon to establish such results?

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1/2

We thank the reviewer for the thoughtful and encouraging review. We appreciate that the reviewer found the problem practical and well-motivated, the Return/Collect/Observe framing intuitive and useful, and the framework agnostic to the bound estimator.


W1. Overclaiming the decomposition.

"... it is just an inner approximation ... the honest framing here should be that the paper provides a one-sided certificate about non-identifiability ..."

Thank you for this insightful observation. You are right that ⋂P∈CIP\bigcap_{P\in C} I_P is an inner approximation of the true non-ID set IP⋆I_{P^\star}; this is exactly what Proposition 1(i) establishes: on the coverage event, ⋂P∈CIP⊆IP⋆\bigcap_{P\in C} I_P \subseteq I_{P^\star}. The paper interprets it this way in the remark after Proposition 1 ("a high-probability inner certificate of population non-identification"), in the Collect rule (Sec. 4.2) and in Appendix A. However, we agree that Definition 7 and parts of the abstract, introduction and conclusion describe it more strongly. We will revise all of these consistently: IP⋆I_{P^\star} is the non-ID region, and ⋂P∈CIP\bigcap_{P\in C} I_P is its high-probability inner approximation.


Q1. Convergence of the intersection.

"In what sense the intersection is the nonID uncertainty, given that it changes with the sample size? Is there a result showing that the intersection converges to IP⋆I_{P^\star} as we get more data?"

The intersection is a data-dependent inner certificate of the fixed population set IP⋆I_{P^\star}. It contains the effect values that are non-identifiable under every candidate P∈CP \in C, so on the coverage event it lies inside IP⋆I_{P^\star} (Proposition 1); with the exact P⋆P^\star (infinitely many samples), it would equal IP⋆I_{P^\star}.

Yes, it converges. As nn grows, CC shrinks around P⋆P^\star and the intersection grows toward IP⋆I_{P^\star}. Eq. (6) (App. D.4) quantifies this: ∣IP⋆∣−4Λrn≤∣⋂P∈CnIP∣≤∣IP⋆∣|I_{P^\star}| - 4\Lambda r_n \le |\bigcap_{P\in C_n} I_P| \le |I_{P^\star}| with rn=O(1/n)r_n = O(1/\sqrt{n}), so the gap vanishes at rate O(1/n)O(1/\sqrt{n}).

Empirically (two-IV SCM, IP⋆=[0.060,0.396]I_{P^\star}=[0.060, 0.396], M=300M=300 candidates, mean over 20 datasets), the intersection grows toward IP⋆I_{P^\star} with both exact closed-form bounds (CF) and the DCM, and stays inside IP⋆I_{P^\star} in every run:

nInner, CFInner, DCM
1kemptyempty
3k[0.198, 0.235][0.196, 0.238]
10k[0.149, 0.316][0.140, 0.333]

From 3k to 10k samples, the CF gap to IP⋆I_{P^\star} shrinks from 0.30 to 0.17, a factor of about 1.8, matching the O(1/n)O(1/\sqrt{n}) rate (10/3≈1.8\sqrt{10/3} \approx 1.8).


W2. Empty intersection.

"... if the effect is identifiable and the confidence set C\mathcal{C} has more than one element ... the intersection of the intervals will be empty ... [or] the identified set is narrow relative to the confidence set (e.g., strong IV settings)."

We completely agree that ⋂P∈CIP=[L‾,U‾]\bigcap_{P\in C} I_P = [\overline{L}, \underline{U}] can be empty (L‾>U‾\overline{L} > \underline{U}) in both cases. The manuscript already shows such cases empirically (Fig. 4 setup 3, Fig. 5 and Fig. 6a), and we will add an explicit remark after Theorems 3 and 7:

  • Identifiable effect: each IP={f(P)}I_P = \lbrace f(P)\rbrace is a single point, so the intersection is empty whenever CC contains candidates with different effects.
  • Non-ID effect with a narrow identified set: when IP⋆I_{P^\star} is narrow relative to the confidence set (4Λrn>∣IP⋆∣4\Lambda r_n > |I_{P^\star}| in Eq. 6), different candidates yield narrow intervals IPI_P covering different ranges of values. These can be disjoint, so their intersection is empty. As nn grows, the confidence set shrinks, the intervals overlap, and the intersection becomes non-empty (Theorem 7).

Q2. When the intersection is empty and what to do.

"Why/when should the intersection ... be non-empty? What should the practitioner do when it is empty?"

When is it non-empty? When the effect is non-ID (∣IP⋆∣>0|I_{P^\star}| > 0) and the confidence set is small relative to ∣IP⋆∣|I_{P^\star}|: by Eq. (6), the intersection contains an interval of width at least ∣IP⋆∣−4Λrn|I_{P^\star}| - 4\Lambda r_n, which is positive once n>N⋆=⌈8Λ2ln⁡(2m/α)/∣IP⋆∣2⌉n > N^\star = \lceil 8\Lambda^2\ln(2m/\alpha)/|I_{P^\star}|^2 \rceil (Theorem 7). Otherwise, it can be empty, as in the two cases of W2.

What should the practitioner do when it is empty? Algorithm 1 returns Return if the outer region excludes 0 (line 11), Observe if the non-ID region is non-empty and contains 0 (line 13), and Collect otherwise (line 15). Since an empty non-ID region cannot trigger Observe, the practitioner collects more samples unless the outer region already excludes 0. For an ID effect, more samples shrink the bounds to a point until Return. For a non-ID effect, the non-ID region becomes non-empty once n>N⋆n > N^\star, after which Observe triggers if it contains 0.

AuthorsOpenReview ↗

Rebuttal Part. 2/2

Q2 (continued). Can the two be distinguished?

"Is it possible to distinguish ... identifiability from ... a narrow non-identified set?"

Two types of information that we have access to during the algorithm execution, can help distinguish an empty intersection due to identifiability from one due to a narrow non-identified set:

  1. From the graph. Since the ADMG is known, identifiability can be checked directly with the ID algorithm [1].
  2. From the candidate bounds. If the effect is ID, LP=UPL_P = U_P for every candidate P∈CP \in C, including those that differ from P⋆P^\star; if it is non-ID, LP<UPL_P < U_P.

W3. Imprecise theory (with Q3 and Q4).

"Definition 3 is imprecise ... Section 2 refers to Theorem 8 ... but the theorem statement is only about identifiable interventional quantities ... Proposition 2 is not proved."

Thank you for the careful reading. These points concern how existing results are stated, and we will fix each in the revision:

  • Definition 3. We agree that "sufficiently high-dimensional" is imprecise. We will cite [3] for a precise characterization of the noise dimension required for discrete variables. Empirically, we use a 64- or 128-dimensional noise vector for each NCM.
  • Theorem 8 reference. We agree that Theorem 8 covers only identifiable quantities. For non-identifiable effects, we follow the neural causal model literature, which bounds an effect by minimizing and maximizing it over neural causal models consistent with the observational distribution, with guarantees for discrete SCMs [2]. Since we assume discrete variables (Assumption 1), the framework of [2], whose neural causal models can express every discrete SCM compatible with GG, applies directly to our setting. We will cite that more explicitly.
  • Finite-state, connected, and Lemma 3 (Q3). A finite-state causal graph has finitely many observed variables, each with finite support (Assumption 1). A connected set of observational distributions is a connected subset of the probability simplex under the Euclidean topology; our confidence sets are continuous images of products of intervals, hence connected. Lemma 3 justifies the connectedness step in the proof of Theorem 2: under its conditions, the response-function parameters consistent with PP form a non-empty, compact and convex (hence connected) set, so the attainable effects form an interval. We will make this step explicit, and define these terms and state the conditions of Lemma 3 with proper context in the main paper.

Q4. Proof of Proposition 2.

"Could the authors provide a proof of Proposition 2?"

Proposition 4 in Appendix D proves the same statement for P(y∣do(x))P(y\mid do(x)), and the ATE case follows identically. We will link Proposition 2 to it, state it precisely (the length of the sample uncertainty region converges to zero as n→∞n \to \infty, given continuity of the bounds at P⋆P^\star), and add the following direct proof. Intuitively, the outer band is at most the variation of the bounds over the confidence set, whose radius rn=O(1/n)r_n = O(1/\sqrt{n}) shrinks with nn (for X→YX \to Y, its width is exactly 2rn2r_n).

Proof. Write IP(x)=[ℓ(P),u(P)]I_P(x)=[\ell(P),u(P)]. Suppose the non-empty confidence sets satisfy sup⁡P∈Cnd(P,P⋆)→p0\sup_{P\in C_n} d(P,P^\star) \xrightarrow{p} 0, and ℓ,u\ell,u are continuous at P⋆P^\star. By continuity, for h∈{ℓ,u}h\in\lbrace \ell,u\rbrace,

sup⁡P∈Cnh(P)−inf⁡P∈Cnh(P)→p0.\sup_{P\in C_n} h(P)-\inf_{P\in C_n} h(P) \xrightarrow{p} 0.

The sampling-uncertainty region

Rn=(⋃P∈CnIP(x))∖(⋂P∈CnIP(x))R_n = \Big(\bigcup_{P\in C_n} I_P(x)\Big) \setminus \Big(\bigcap_{P\in C_n} I_P(x)\Big)

is contained in

[inf⁡P∈Cnℓ(P), sup⁡P∈Cnℓ(P)]∪[inf⁡P∈Cnu(P), sup⁡P∈Cnu(P)].\Big[\inf_{P\in C_n}\ell(P),\ \sup_{P\in C_n}\ell(P)\Big] \cup \Big[\inf_{P\in C_n}u(P),\ \sup_{P\in C_n}u(P)\Big].

Hence its total length converges to zero:

∣Rn∣≤(sup⁡P∈Cnℓ(P)−inf⁡P∈Cnℓ(P))+(sup⁡P∈Cnu(P)−inf⁡P∈Cnu(P))→p0.|R_n| \le \Big(\sup_{P\in C_n}\ell(P)-\inf_{P\in C_n}\ell(P)\Big) + \Big(\sup_{P\in C_n}u(P)-\inf_{P\in C_n}u(P)\Big) \xrightarrow{p} 0.


Q5. Choosing the confidence radius.

"If we are interested in statements like 'with probability at least 1−δ1-\delta', how can one choose the radius ϵ\epsilon?"

We pick the target confidence 1−δ1-\delta and set the radius of each of the mm conditional probabilities by Hoeffding's inequality with a Bonferroni correction (Sec. 4.3, App. E.3):

ϵi=ln⁡(2m/δ)2ni,\epsilon_i = \sqrt{\frac{\ln(2m/\delta)}{2n_i}},

where nin_i is the number of samples for parent configuration ii. By a union bound, P⋆∈CP^\star \in C with probability at least 1−δ1-\delta (our α=δ\alpha = \delta), so our guarantees hold with probability at least 1−δ1-\delta.


We would be happy to have further discussion. If we have addressed all the concerns, we kindly request that the reviewer to consider increasing their score.

References

[1] Identification of joint interventional distributions in recursive semi-Markovian causal models.

[2] The causal-neural connection: Expressiveness, learnability, and inference.

[3] Partial counterfactual identification from observational and experimental data.

Dear Reviewer WWJR,

Thank you very much for your time and effort in reviewing this submission. The authors have now submitted their rebuttal, and the discussion period is nearing its end. We would greatly appreciate it if you could take some time to carefully review the authors’ responses and share your assessment of whether your concerns have been adequately addressed. Thank you again for your valuable time and contributions to the review process.

Dear Reviewer WWJR,

Thank you for your detailed review and positive assessment of our paper. If you have any remaining concerns, please let us know, and we would be happy to address them.

If you feel our responses have addressed your concerns, we would be grateful if you could support our paper during the Reviewer-AC discussion period.

Sincerely,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.