- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper discusses the difficulties of CATE estimation and uncertainty quantification in the few-placebo regime. As existing estimators are prone to undercoverage, the paper introduces ISO-HMC, a GP model that incorporates a specific treatment effect prior. The method propagates the uncertainty through both the outcome model and the treatment effect hyperparameters, resulting in calibrated CATE intervals.
Critical strengths
- Setting: The few-placebo regime is relevant for medical applications
Critical weaknesses
- Novelty: ISO is the BCF parameterization (Hahn et al., 2020) with GPs and is nearly identical to the paper’s own BCF-GP baseline. HMC marginalisation of GP hyperparameters is standard practice. Much of Sections 2–4 restates known results on non-orthogonal plug-in bias and overlap sensitivity and is often redundant
- Identification: The paper seems to confuse identification with estimation difficulty. It calls “weakly identified,” but it is identified in a standard fashion under the stated assumptions. Furthermore, the few placebo regime does not directly result in poor overlap (as the paper incorrectly states and motivates)
- Correctness: In RCTs, the propensity score is known. Thus, R(X) in Eq. 6 is exactly 0, making the argument about the remaining bias and Eq. 7 wrong and unnecessary. Furthermore, the amplification in line 224 should be of order .
- Guarantees: No coverage guarantee is provided. The meaning of calibration in the experiments is unclear.
- Experiments: In general, the experiments do not necessarily support the claims of the paper (e.g., undercoverage).
- Related work: The paper only introduces the baselines. Bayesian CATE models and literature on coverage of Bayesian credible sets are not discussed.
- Clarity: Overall, the structure is hard to follow: results appear before the setup, and the conformal prediction paragraph comes without context. Furthermore, the mathematical notation is not defined before its usage and there are many rendering errors and broken references.
Questions for authors
- What does “weakly identified” mean here, given that is identified?
- With known propensity, we have that R(X) = 0. Why does Table 3 report such bias in the experimental results?
- How does ISO-HMC differ from BCF/BCF-GP beyond applying HMC?
- Can you provide any coverage guarantee? Maybe asymptotically or under correct specification?
- Why were these baselines chosen? This should be justified. Also, why are conformal CATE methods and BCF not benchmarked?