- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CausalGuard studies treatment-effect inference when the adjustment set is uncertain. It proposes candidate causal graphs using LLM-derived edge priors, screens edges using conditional-independence tests, and assigns graph weights using BIC and the prior. Graphs sharing an adjustment set share a downstream estimator. The method averages adjustment-specific AIPW pseudo-outcomes and preliminary interval endpoints, then applies split-conformal calibration to the aggregate. It establishes marginal predictive coverage for the aggregated pseudo-outcome and connects its conditional mean to CATE under additional alignment, nuisance-estimation, and weight-concentration assumptions. Experiments include causal benchmarks, collider stress tests, and component ablations.
Critical strengths
-
The paper addresses a relevant concern: uncertainty about adjustment variables can introduce causal bias that predictive calibration alone cannot repair. Sharing estimators across graphs with identical adjustment sets is also a sensible computational design.
-
The paper explicitly distinguishes pseudo-outcome coverage from empirical CATE coverage and includes graph-aware controls, Top-1 comparisons, prior perturbations, and multiple LLM backbones. These comparisons help assess which components contribute to the results.
Critical weaknesses
-
The inferential objective does not clearly resolve the motivating problem. The guarantee covers a constructed pseudo-outcome, not the causal effect. Convergence of its conditional mean to CATE does not establish CATE confidence-interval coverage. Causal interpretation additionally requires weights to concentrate on target-aligned strategies, but the paper does not establish why its LLM/BIC procedure achieves this under observational ambiguity. The pipeline makes the identifying assumptions harder to interpret without clearly resolving identification uncertainty.
-
The novelty and empirical value of the additional components remain unclear. Once the aggregate is fixed using training data, calibration follows standard split-conformal reasoning, while score domination follows from convexity. Empirically, Full and Top-1 have nearly identical widths on several datasets. On Jobs, Uniform BMA has similar width and coverage; in the IHDP prior comparison, uniform proposals have lower RMSE than the LLM prior (.859 versus .904). Four LLM backbones yield identical reported width and RMSE, which may indicate limited LLM influence rather than a demonstrated benefit. The collider result is promising, but does not isolate the necessity of LLM guidance or aggregation. Several central comparisons lack uncertainty estimates for paired differences.
Questions for authors
-
Why is prediction of the aggregated pseudo-outcome the appropriate scientific objective, rather than confidence intervals for the causal effect? What guarantee addresses the causal estimand when adjustment uncertainty remains unresolved?
-
What interpretable identification assumptions justify the aggregate, and what establishes that the proposed weights favor target-aligned strategies when different graphs fit the observational data equally well?
-
What is technically new beyond ensembling followed by split-conformal calibration, and what matched empirical evidence establishes benefits over Top-1 selection and uniform proposals?
-
Do different LLMs meaningfully change adjustment sets or weights? Do the nearly identical results demonstrate robustness, or little influence from the LLM?