- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper develops causal decision making framework under partial identification. Finite sample uncertainy is separated from non-identification irreducible (modelling) uncertainty by taking intersections and unions of partially identified effect sets over a confidence set of observational distributions. This yields a rule for deciding whether to act, collect more samples, or observe additional variables. The resulting nested optimization is approximated using deep causal models (UA-DCM) and evaluated on synthetic and real-world data.
Critical strengths
Problem formulation and presentation: I appreciate the clear exposition throughout the paper. Paper is clearly written and easy to follow Useful connection between partial identification and data collection. The framework turns identification bounds into actionable statements which seems clearly useful to me.
Critical weaknesses
-
Limited methodological novelty / adaptive decision making: The core construction is a direct application of finite-sample uncertainty quantification on top of partial identification: construct a confidence set over observational distributions, obtain an identified set for each, and take their intersection/union. The more interesting opportunity and a natural extension is adaptive data acquisition, but this is not developed. "OBSERVE" diagnoses that additional variables are required without deciding which variables should be acquired. I would have liked this developed further to deepen the novelty and contribution.
-
Theory-to-algorithm gap and insufficient validation: The guarantees apply to exact optimization over the confidence set and exact causal-effect bounds, while UA-DCM approximates both using finitely sampled candidate distributions and non-convex neural optimization. The experiments do not establish how accurately UA-DCM recovers the ideal decomposition or its resulting decisions. In the synthetic settings where closed-form bounds are available, the paper can and should separate the ideal decomposition, confidence-set approximation, and DCM approximation and evaluate endpoint and RETURN/COLLECT/OBSERVE decision errors across repeated datasets. Looking at the experiments the approximation seems to sometimes struggle quite a bit compared to baselines (fig 4) - I would want to understand if the issue is the complexity of implementation or possibly in the theory.
Questions for authors
-
In the synthetic settings with closed-form causal bounds, how closely does UA-DCM recover the exact inner/outer decomposition? In particular, across repeated datasets, how often does it produce the same RETURN/COLLECT/OBSERVE decision?
-
Can the framework be extended to choose adaptively which additional variable or information source to acquire, rather than only concluding that additional variables should be observed?
6 comments
Rebuttal Part 2/2
W2 / Q1. Separating the error sources.
"... should separate the ideal decomposition, confidence-set approximation, and DCM approximation and evaluate endpoint...across repeated datasets."
To address the reviewer's concern, we use the closed-form bound as a baseline and compare it with UA-DCM. This separates the ideal decomposition and the confidence-set approximation from the DCM approximation. Below we discuss three sources of error.
- Bound error. For the bow and IV graphs, the Balke-Pearl bounds are exact and tight, so each candidate's is computed without error. The DCM, in contrast, approximates the bound for any graph and distribution; it is trained to match the observed distribution only approximately (within a small total variation distance), so it can reach slightly more extreme effects than an exact fit, and its bounds are about 0.015 to 0.02 wider per end.
- Confidence-set coverage. We use the Hoeffding-Bonferroni set, which contains with probability at least and is conservative in practice. At , in every setting, the outer region contained and the inner region stayed inside in 100% of datasets, so it is not a source of error.
- Number of candidates . We extended the Fig. 4 experiment to and 300 candidates explored in the -ball. With small , the inner region is wider than the correct inner region and the outer region is narrower than the correct outer region. Below we show the evaluation.
Endpoints across repeated datasets. We compare the closed form (CF) and the DCM on the same datasets and candidates (both instruments, , 20 replicates). At fixed , the DCM closely matches CF's endpoints and takes the closed form's decision in 98% of runs at (87% at ).
| N | Outer, CF | Outer, DCM | Inner, CF | Inner, DCM |
|---|---|---|---|---|
| 1k | [-0.192, 0.591] | [-0.199, 0.599] | Empty | Empty |
| 3k | [-0.102, 0.503] | [-0.117, 0.516] | [0.198, 0.235] | [0.196, 0.238] |
| 10k | [-0.025, 0.463] | [-0.046, 0.481] | [0.149, 0.316] | [0.140, 0.333] |
| 1k | [-0.094, 0.502] | [-0.099, 0.508] | [0.175, 0.254] | [0.173, 0.254] |
| 3k | [-0.055, 0.439] | [-0.066, 0.452] | [0.128, 0.291] | [0.120, 0.296] |
| 10k | [+0.005, 0.431] | [-0.014, 0.449] | [0.105, 0.344] | [0.088, 0.362] |
"Empty" means maxmin exceeds minmax .
"and RETURN/COLLECT/OBSERVE decision errors"
Decision errors across repeated datasets. We re-ran the Fig. 4 scenario sequentially (). The analyst starts with only. Observe adds the next instrument (, then ) for the same units; Collect increases . Each cell reads action (N = sample size). For both methods, every Return has the correct sign and every Observe occurs where .
Evaluation 1. Ai is the -th action (; 20 replicates; % is the share of runs following that trajectory). The following table shows the most frequent trajectory across 20 independent replicates.
| M | Method | % | A1 | A2 | A3 | A4 | A5 | A6 | A7 | A8 | A9 | A10 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 300 | CF | 65% | Observe I1 (N=30) | Collect (N=30) | Collect (N=100) | Collect (N=300) | Observe I2 (N=1k) | Collect (N=1k) | Collect (N=3k) | Collect (N=10k) | Return (N=30k) | |
| 300 | DCM | 50% | Observe I1 (N=30) | Collect (N=30) | Collect (N=100) | Collect (N=300) | Observe I2 (N=1k) | Collect (N=1k) | Collect (N=3k) | Collect (N=10k) | Collect (N=30k) | Return (N=39k–51k) |
Evaluation 2. For each step, the table below gives the % of runs whose action at that step matches the reference path (the most common closed-form path at ):
| A1 | A2 | A3 | A4 | A5 | A6 | A7 | A8 | A9 | A10 | |
|---|---|---|---|---|---|---|---|---|---|---|
| Reference | Observe I1 (N=30) | Collect (N=30) | Collect (N=100) | Collect (N=300) | Observe I2 (N=1k) | Collect (N=1k) | Collect (N=3k) | Collect (N=10k) | Return (N=30k) | ended |
| M | Method | A1 | A2 | A3 | A4 | A5 | A6 | A7 | A8 | A9 | A10 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 300 | CF | 100 | 100 | 100 | 95 | 70 | 75 | 100 | 95 | 90 | 95 |
| 300 | DCM | 80 | 80 | 100 | 95 | 55 | 60 | 95 | 100 | 10 | 10 |
| 10 | CF | 100 | 95 | 90 | 40 | 40 | 90 | 80 | 30 | 30 | 100 |
| 10 | DCM | 100 | 100 | 85 | 35 | 30 | 85 | 85 | 55 | 35 | 80 |
The DCM leaves the reference path mainly at A9: its bounds are slightly wider (approximate fit), so it returns one step later at N= 39k–51k. Every run of both methods returns with a correct action. We will add the full experimental setup with further details (e.g., ) to the paper.
We are happy to address any further concerns. If we have addressed all, we kindly request the reviewer to consider increasing the score.
Thank you for the detailed response and additional experiments. I appreciate the effort, and the new results substantially improve the empirical validation of the proposed approach.
Nevertheless, my overall assessment of the contribution remains largely unchanged. While I find the decision-oriented framing practically useful, the core methodology remains a relatively straightforward application of established ideas from partial identification and finite-sample uncertainty quantification. In my view, the paper offers limited additional methodological or conceptual insight, and I do not find the overall research contribution sufficiently substantial (for ICLR).
I therefore maintain my score of 2 (Weak Rejection).
Rebuttal Part 1/2
We thank the reviewer for the constructive review. We highly appreciate that the reviewer found the paper clearly written and easy to follow, the connection between partial identification and data collection useful, and the actionable statements clearly useful. Below, we address their concerns.
W1. Novelty.
Our main contribution is a finite sample decision framework that separates sampling uncertainty from identification uncertainty and connects it to the decision-making problem. We agree that the construction is conceptually simple. However, it is important and novel for decision-making problems, and has not been studied before. Moreover, obtaining and exploring informative confidence regions for the causal bounds is not trivial in general.
W1 / Q2. Adaptive data acquisition.
We agree with the reviewer that adaptive data acquisition is an interesting next step. Selecting which variables to acquire raises several challenges:
Formally characterizing these challenges and addressing them is a substantial separate research problem, which we leave to future work.
Nonetheless, a few directions are:
Fig. 4. Implementation or theory.
The gap in Fig. 4 is an approximation error in the implementation, not a problem with the theory, and it is fixed by simple hyperparameter tuning. The DCM estimates the data fit and the ATE with Monte Carlo latent draws. The original results used 256 draws per step; we re-ran with 4096 draws per step, and the DCM now closely matches the closed form (the same decision in 98% of runs at M=300). Moreover, prior work shows theoretically that bounds obtained by optimizing over sufficiently expressive neural causal models recover the true bounds [2, 3].
References
[1] Identification of joint interventional distributions in recursive semi-Markovian causal models.
[2] Partial identification of treatment effects with implicit generative models.
[3] The causal-neural connection: Expressiveness, learnability, and inference.