- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
In causal inference, hidden confounding could bias the treatment effect estimation and is very common in real-world practice. This paper propose ProCI (Progressive Confounder Imputation) framework, which use large langue models (LLMs) to address this challenge by providing additional informaton. This framework is built on two assumptions (line 19-22) i) LLM can interpret rich text information in causal benchmarks; ii) LLM has world knowledge that allows generating latent confounders and support causal reasoning.
ProCI estimates treatment effect in two steps, confounder imputation and unconfoundness validation (Section 3). In confounder imputation phase, ProCI let LLM generate hidden confounder U based on the semantic relationship among observed covariate X, outcome Y, and treatment T, along with a text description. LLM is then use to impute the value of U by inferring its distribution type and parameters. After imputation, the observed confounder X is then augmented by incorporating U. After phase 1, ProCI performs an conditional independence test to assess whether the generated confounder U captures sufficient hidden bias. If the test fails, it returns to Phase 1 to generate an additional confounder; if it passes, it proceeds to CATE estimation.
The authors then conduct experiments to answer research questions (RQs) 1 to 5 (line 305-311). In RQ1, they show ProCI can improve the performance of existing CATE estimatiors on the evaluated dataset. In RQ2, ProCI demonstrated better improvement with domain-adaptive LLM routing. In RQ3, the authors show ProCI gives higher conditional mutual information (CMI) than random. In RQ4, the authors testes the sensitivity of ProCI to the number of hidden confounders. In RQ5, the author did an ablation study to evaluate the contribution of components of ProCI.
In conclusion, this paper propose ProCI, a framework to mitigate hidden confounding in treatment effect estimation.
Critical strengths
-
Important research question. This paper studies how to address hidden confounding in treatment effect estimation. It is a fundamental problem in causal inference and addressing that can reduce bias in estimation and better inform the final decision making. Exploring whether LLMs encode hidden confounding information and how to use that information is an interesting research direction.
-
Clear writing. The paper is well structured. I can easily follow the reasoning flow and understand the proposed framework. The progressive imputation procedure is easy to understand and well explained. The paper also provide many experiment to support the argument, including many CATE baselines, different LLM models, and ablation studies on the level of hidden confounding and components of the proposed method.
Critical weaknesses
-
The statistical validity of LLM-generated confounder in phase 1 is not sufficiently generated. This paper propose using LLM to generate hidden confounding. Specifically, in equation (2), . In addition, ProCI also let LLM infer the distribution and its parameter, in Equation (3) and (4), respectively. Therefore, the proposed hidden confounder not only depend on observed covariate , but also on treatment and outcome . This causes severe identification issue, since a valid adjustment variable needs to precede treatment and outcome. The paper lacks sufficient justification that the imputed variables belong to valid adjustment set. As a result, this procedure may lead to strong bias. Moreover, the paper assumes LLM can provide correct information for hidden confounding. However, LLM may hallucianate and the paper does not give sufficient discussion on the consequence of that.
-
The unconfoundedness validation in phase 2 is not neccessarily correct. To test whether LLM generated confounders are valid, the paper proposes using conditional independence test . To address the problem that the counterfactual outcomes, or , are never observed, the authors use LLM to impute the missing outcomes, in Equation (5). Assuming LLM can correctly impute unobserved outcomes is a very strong assumption, which is unlikely to hold in practice.
Even though we assume that LLMs may correctly impute the the unobserved outcomes, the conditional independence test cannot justify that generated confounders are valid adjustment set in general. For example, let , where is some measurable function. Then even is not a valid adjustment variable. Conditioning on will break the positivity assumption.
Questions for authors
-
Under what conditions can LLM provide valid adjustment set rather than variables inferred post-hoc from and ? Furthermore, can we verify these conditions in real-world application? If so, how?
-
Under what conditions can the proposed unconfoundedness validation correctly verify that current covariates are sufficient for identification? If the conditional independence test in Equation (6) is needed for that, please justify why LLM give correct imputation for the unobserved outcomes or .