- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces proximal balancing, which learns representations and audits conditional balance using held-out proxies to identify average treatment effects under unmeasured confounding. PROBE handles unknown valid proxies through screening and aggregation. The paper provides identification and error analyses, supported by synthetic, semi-synthetic, and observational evaluations.
Critical strengths
-
The method combines proxy balancing with representation learning without explicitly solving a bridge equation or recovering the latent confounder, offering a valuable new approach to proxy-based causal inference.
-
The analysis covers exact identification, approximate balance, effect estimation, and aggregation with unknown valid proxies, forming a well-developed theoretical and algorithmic framework.
-
The authors explicitly acknowledge limitations of the balance audit, image experiments, and uncertainty estimates, helping readers interpret the empirical evidence appropriately.
Critical weaknesses
-
The comparison with standard proximal assumptions is useful, but I find the assumptions difficult to relate to real measurements. It is unclear when the retained proxies contain enough information while the held-out proxies can still reveal hidden differences that affect the outcome.
-
The method relies on informative proxies and on correct estimates forming the largest group. The experiment tests one invalid candidate, but offers limited evidence about weaker proxies or several incorrect estimates agreeing with each other.
-
The image experiment uses a frozen, supervised reader to recover scalar values before applying PROBE. This makes it hard to judge how well the method handles image proxies when useful information cannot be extracted accurately.
Questions for authors
-
Can a concrete measurement example explain when outcome-relevant completeness and balance attainability hold together?
-
How does PROBE perform when Assumption 4 fails, especially when proxies become less informative or several incorrect estimates agree?
-
How does PROBE perform when the image reader makes errors or the images contain more complex proxy information?
3 comments
Response to Reviewer vP9x (2/2)
Can a concrete measurement example explain when outcome-relevant completeness and balance attainability hold together?
In SCM-1 (Section 5, Figure 8(a)), W₁, W₂, and W₃ are independent noisy measurements of U. If W₁ is held out, the retained W₂ and W₃ carry information about U, so balance is attainable. W₁ is independent of treatment given U and is a measurement of U with independent noise, so outcome-relevant completeness holds. In contrast, W₅ also affects treatment and violates held-out independence, and W₄ is shifted by W₅ and violates completeness. Appendix B.1 proves that both conditions hold in a simplified Gaussian measurement model of this kind, in which Var(U | Z) = 1/6, so balance does not require recovering U (pp. 17–18).
How does PROBE perform when Assumption 4 fails, especially when proxies become less informative or several incorrect estimates agree?
Incorrect estimates exist, but valid splits form the largest group. PROBE returns a valid estimate, as in the stress test above with only W₁ and W₂ valid: 8 of 10 returned, all from the valid group.
Incorrect estimates agree and form the largest group. PROBE returns a wrong estimate. When W₂, W₃, and W₄ were invalid in addition to W₅, so W₁ was the only valid block, the splits holding out the invalid blocks W₃ and W₄ gave similar biased estimates and formed the largest group, and PROBE returned 0.553 against a true effect of 1 (Appendix D.6, Table 3, p. 33). This failure is not visible from the output.
Proxies become less informative. When we raised the measurement-noise standard deviation of every proxy in SCM-1 from 0.85 to 1.1, 1.4, and 1.7, PROBE returned in every dataset, and its MAE was 0.021, 0.024, 0.135, and 0.249 (Appendix D.15, Table 10, p. 46). Weaker proxies capture less of U, so the error grows. In terms of Assumption 4, the same hidden difference changes the held-out proxy less, which corresponds to a larger stability constant.
How does PROBE perform when the image reader makes errors or the images contain more complex proxy information?
Reader errors. We added Gaussian noise with standard deviation 0.15 to the Shapes3D pixels and kept the reader frozen. The reader's orientation-level error rose from 0% to 23.5%, yet PROBE's MAE was 0.0172, against 0.0194 on clean images, over ten datasets (Appendix D.14, Table 9, p. 45).
More complex images. In the chest X-ray study, when one region shows a mediator of the treatment effect, adjustment for the whole image has MAE 0.260, while PROBE hides that region and has MAE 0.149. When a treatment-response mark appears in every region, hiding a region cannot help (MAE 0.436, against 0.390 for the whole image) (Appendix D.9, Table 6, p. 41).
Thank you for taking the time to respond to each of my questions in detail. Your responses have helped clarify most of my concern. After considering the responses, I will keep my score unchanged.
Response to Reviewer vP9x (1/2)
Thank you for the insightful review.
A held-out proxy is useful when it can be regarded as independent of treatment given the hidden confounder, for example a pretreatment measurement that the treatment decision does not use. In a chest X-ray, severity U shows in every block of the image (four lung quadrants and a central strip) and drives treatment. A region without treatment-related content, such as a device or a treatment response, is independent of treatment given U, yet it still shows severity, which affects the outcome. The other regions also show severity, so they carry enough information. In our chest X-ray study, the five region features explain 91% of the variance of U, and a region with a treatment-dependent mark breaks this condition and is flagged by the screen (Appendix D.9, pp. 38–41).
We added a stress test in which the valid splits are not a majority. In ten SCM-1 datasets in which only W₁ and W₂ are valid and W₃, W₄, and W₅ are invalid, PROBE returned an estimate in 8 of 10, all from the valid group (MAE 0.0309). In 7 of these 8, the valid group held exactly half of the retained splits (group sizes 3:2:1), so the valid group only needs to be the largest group, not a majority. In the other 2, the valid and invalid groups had the same size, and PROBE declined to return an estimate (Appendix D.6, pp. 33–34).
Our chest X-ray study addresses this. It uses no scalar reader: each of five image blocks is encoded by a network trained on that block alone, with cross-fitting, giving 80 features that explain 91% of the variance of severity U. When a treatment-response mark appears in one region, adjustment for the whole image has MAE 0.268, while PROBE hides that region and has MAE 0.138, close to the 0.132 of unmarked images. That 0.132 is the part of U the encoder cannot extract, which no choice of blocks can remove (Section 5, pp. 9–10, Figure 5; Appendix D.9, Table 6, p. 41).