- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLMs can assign covariates to the roles required by proximal causal inference (PCI): treatment proxies (Z), outcome proxies (W), observed confounders (X), or discarded variables (D). Given the PCI assumptions, study context, and variable descriptions, an LLM infers the latent confounder and proposes a partition, while the downstream bridge estimator is held fixed. The paper also introduces stratum-balanced bridge training to estimate conditional effects across observed strata. Experiments cover SUPPORT, Framingham Offspring, and a 10-million-sample food-delivery dataset evaluated against a separate randomized price experiment.
Critical strengths
The presentation of this paper is clear and easy to understand. The experiments are comprehensive by incorporating the randomized price experiment.
Critical weaknesses
The paper does not clarify how semantic plausibility from variable descriptions translates into PCI-valid proxy roles. In specific, A2/A3 require conditional independence given latent, and A4 requires completeness. Moreover, Framingham is relatively insensitive to proxy reassignment once is fixed, leaving the added value over general confounder identification unclear.
Questions for authors
Given A2–A4, how do the authors distinguish an LLM’s semantically plausible inference about (U) from evidence that the selected (Z/W) satisfy the identification conditions? What criterion can validate a partition beyond the plausibility of its causal explanation?
We thank the reviewer for a thoughtful review that goes to the paper's central question. How plausibility relates to the identification conditions is the right question, and we try to answer it directly.
What cannot be verified. A2 and A3 are conditional independences given the latent U, and A4 is a completeness condition on the same latent variable; none is testable from observed data in general. This is the proximal counterpart of the untestability of ignorability, and it applies equally to every source of a partition: to the expert who supplied the SUPPORT allocation, to an association rule, and to a language model. So, to the reviewer's direct question, we know of no criterion that certifies a partition, and we will say this explicitly in Section 3 and the limitations. The paper's design follows from accepting it. The partition's only observable consequence is the estimate, so we hold the estimator fixed and judge each partition by whether its estimate agrees with a reference the model never saw: published confidence intervals on SUPPORT, a range bracketed by randomised trials on Framingham, and a separate randomised experiment on the waybill corpus. Random and perturbed partitions through the same pipeline show how far a wrong partition can move the estimate, so agreement with the reference is informative rather than automatic.
What can be checked. Two necessary conditions are testable from data: Z must carry information about U, which appears as first-stage strength, and Z and W must be dependent given X, since both are driven by U. These can rule a proxy out but cannot rule it in, and they cannot tell dependence through U from a direct channel between Z and W. The pH and PaCO2 case in Section 5.1 shows the limit: a first-stage F of about 1,100 says the two are tightly bound, but whether the binding runs through U or through the Henderson-Hasselbalch identity is a question of chemistry, not of the data, which is why the paper reports the models' reasoning on that pair alongside the statistic. We will state both checks and their limit in the revision.
Beyond plausibility. Four observations bear on whether the models do more than produce a plausible story. First, Section 5.3: a judgment based on influence cannot produce a treatment proxy, because a valid Z does influence the outcome through U, and the six cardiologists left that role empty on both cohorts. Every admissible LLM vote fills it, with reasoning of the form A3 requires, no direct path to Y once U is fixed; the one vote with an empty Z (Claude Haiku 4.5 on Framingham) is reported as a completeness failure. This is evidence about the kind of reasoning, not that A3 holds. Second, Section 5.1: six of eight arms move arterial pH from the published W side to Z beside PaCO2, treating the two as readings of one acid-base state that should not sit on opposite sides; arms raise the Henderson-Hasselbalch relation unprompted, and Claude Opus 5, which keeps the published allocation, argues that given U the coupling is loose enough for pH to remain in W (Appendix A). The arms disagree about this pair, but about the right thing: the independence of Z and W given U and X, a condition on the pair (A3b in the prompt, part of A2 in Section 3), not the plausibility of either variable alone. Third, Appendix C.3: on SUPPORT, exchanging Z and W moves six of the eight allocations by 0.3 to 1.7 days, so the side matters on this cohort, and the models' choices keep all eight arms inside the published confidence intervals. Fourth, memorisation does not explain the Opus match: with variable names replaced by shuffled codes and the study's name and counts removed, Claude Opus 5 still returns the published allocation in all five replicates.
Framingham. We agree, and the paper reaches the same conclusion, though it buries it: with each partition's X held fixed, 27 of 30 random reassignments of the proxies stay in the trial range (Appendix C.2), so on this cohort the models' contribution is the choice of observed confounders, and the proximal step adds only a modest shift toward the HOPE-3 value of 0.75: the overlap-weighted Cox fit with all sixteen variables and no proxies gives 0.90, the LLM partitions 0.76 to 0.89. We will say so at the start of Section 5.2 and in the abstract. The proxy-specific evidence rests on SUPPORT and the waybill corpus, where the three treatment-proxy sets are disjoint and the size of the correction differs with them (Table 7). What Framingham does show is that the confounder decision is load-bearing: dropping four of an arm's chosen confounders moves the median hazard ratio to 0.94 with 9 of 20 perturbed partitions in range (Table 10).
The reviewer's distinction between a plausible account of U and evidence that the conditions hold is the one the revision should draw, and we will restructure Section 3 around it.