acceptodds
Under review as a conference paper at ICLR 2027

Calibrated Inference for the Conditional Average Treatment Effect in the Few-Placebo Regime via Gaussian Processes

Abstract

In the few-placebo regime (), the control-arm outcome surface is only weakly identified, and this under-identification is a primary determinant of interval reliability for the conditional average treatment effect (CATE). Existing estimators can under-cover because their inference procedures do not propagate this under-identification into the interval: point-estimate methods inherit nuisance bias, and GP methods with MAP hyperparameter estimation suppress residual uncertainty in the treatment-effect hyperparameters. We propose ISO-HMC, a structured Gaussian-process model that places a separate prior on the treatment-effect function and marginalises over its hyperparameters via HMC. By propagating this uncertainty through the law of total variance, ISO-HMC achieves calibration when the treatment effect is smooth and low-dimensional. We test three claims: (i) ISO-HMC is calibrated in its target regime; (ii) it achieves this without sacrificing interval informativeness; (iii) it degrades gracefully under misspecification, with an independent-arm GP as a safe fallback. All three are supported: on smooth low-dimensional designs ISO-HMC attains calibrated coverage and substantially narrower intervals than a conservative independent-arm GP baseline, while on nonlinear or high-dimensional designs it under-covers and the independent-arm GP restores calibration. Overall, these results show that in the few-placebo regime, calibrated CATE intervals benefit from propagating uncertainty through both the outcome model and the treatment-effect hyperparameters. Suppressing either through point estimates or MAP leads to overconfidence and under-coverage.

Primary area:
causal reasoning
Keywords:
causal inference, treatment effects, uncertainty quantification, Gaussian processes, Hamiltonian Monte Carlo, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qjjJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper discusses the difficulties of CATE estimation and uncertainty quantification in the few-placebo regime. As existing estimators are prone to undercoverage, the paper introduces ISO-HMC, a GP model that incorporates a specific treatment effect prior. The method propagates the uncertainty through both the outcome model and the treatment effect hyperparameters, resulting in calibrated CATE intervals.

Critical strengths
  • Setting: The few-placebo regime is relevant for medical applications
Critical weaknesses
  • Novelty: ISO is the BCF parameterization (Hahn et al., 2020) with GPs and is nearly identical to the paper’s own BCF-GP baseline. HMC marginalisation of GP hyperparameters is standard practice. Much of Sections 2–4 restates known results on non-orthogonal plug-in bias and overlap sensitivity and is often redundant
  • Identification: The paper seems to confuse identification with estimation difficulty. It calls μ0\mu_0 “weakly identified,” but it is identified in a standard fashion under the stated assumptions. Furthermore, the few placebo regime does not directly result in poor overlap (as the paper incorrectly states and motivates)
  • Correctness: In RCTs, the propensity score is known. Thus, R(X) in Eq. 6 is exactly 0, making the argument about the remaining bias and Eq. 7 wrong and unnecessary. Furthermore, the amplification in line 224 should be of order 1/(1−π)1/(1-\pi).
  • Guarantees: No coverage guarantee is provided. The meaning of calibration in the experiments is unclear.
  • Experiments: In general, the experiments do not necessarily support the claims of the paper (e.g., undercoverage).
  • Related work: The paper only introduces the baselines. Bayesian CATE models and literature on coverage of Bayesian credible sets are not discussed.
  • Clarity: Overall, the structure is hard to follow: results appear before the setup, and the conformal prediction paragraph comes without context. Furthermore, the mathematical notation is not defined before its usage and there are many rendering errors and broken references.
Questions for authors
  • What does “weakly identified” mean here, given that μ0\mu_0 is identified?
  • With known propensity, we have that R(X) = 0. Why does Table 3 report such bias in the experimental results?
  • How does ISO-HMC differ from BCF/BCF-GP beyond applying HMC?
  • Can you provide any coverage guarantee? Maybe asymptotically or under correct specification?
  • Why were these baselines chosen? This should be justified. Also, why are conformal CATE methods and BCF not benchmarked?
Reviewer_hhMtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies uncertainty quantification for the Conditional Average Treatment Effect (CATE) in the few-placebo setting, where N0≪N1N_0 \ll N_1. The authors argue that limited control observations make the control outcome function μ0(x)\mu_0(x) difficult to estimate reliably, which leads to biased CATE estimates and poorly calibrated uncertainty intervals. By analyzing the limitations of existing CATE estimators such as the X-Learner and doubly robust approaches, the paper shows how nuisance estimation bias and poor overlap can lead to undercoverage. For this issue, the authors propose ISO-HMC, a structured Gaussian process model of the form Yi=μ0(Xi)+Wiτ(Xi)+ϵiY_i = \mu_0(X_i) + W_i\tau(X_i) + \epsilon_i, where independent Gaussian process priors are placed on the outcome function μ0\mu_0 and treatment effect function τ\tau. The proposed method uses Hamiltonian Monte Carlo (HMC) to marginalize over treatment effect kernel hyperparameters and noise variance.

Experiments on synthetic datasets and the semi-synthetic IHDP benchmark compare ISO-HMC with some existing CATE estimators. The results suggest that ISO-HMC achieves well-calibrated intervals with relatively narrow widths when the treatment effect function is smooth and low-dimensional. However, its coverage seems to behave less superior in more complex or high-dimensional settings, where independent-arm GP-CATE provides more reliable calibration with wider intervals though. The paper further studies how calibration depends on the control-sample-to-dimension ratio N0/dN_0/d, treatment effect complexity, and correlation between treatment and control outcome functions.

Critical strengths
  1. I personally find that this paper identifies an important research problem. The paper addresses uncertainty quantification for CATE estimation under severe treatment-arm imbalance (N0≪N1N_0 \ll N_1). This is a practically relevant but challenging setting.

  2. Analysis of existing methods: The paper provides useful insights into the limitations of existing CATE estimators, particularly regarding nuisance estimation bias, poor overlap, and interval undercoverage and discuss some existing methods on these issues. The paper is thus well motivated.

  3. Empirical evaluation: The paper compares multiple CATE estimators across different settings, highlighting the trade-offs between interval calibration and width, as well as the conditions under which different methods perform well.

Critical weaknesses

I think the main weakness overall of this paper is its depth given I already stated in the strength that this paper's motivation is good.

  1. Limited methodological novelty: The proposed ISO-HMC method mainly combines independent Gaussian process priors with HMC-based hyperparameter marginalization, both of which are well-established techniques. The paper does not clearly demonstrate a substantial methodological advance beyond existing Bayesian GP approaches.

  2. Limited theoretical guarantees: Although the paper discusses calibration failures of existing CATE estimators, it does not establish new theoretical guarantees for the frequentist coverage or calibration of the proposed ISO-HMC credible intervals, particularly in the few-placebo regime. I think for identifying such an important issue (few-placebo setup), some necessary theoretical results would make the paper more solid.

  3. Insufficient empirical evidence for the proposed method: The experiments do not consistently demonstrate a substantial advantage of ISO-HMC over ISO-MAP or other baselines. For example, on IHDP, ISO-HMC achieves coverage of 0.76 compared with 0.75 for ISO-MAP, both substantially below the nominal 0.95 level. More extensive studies and comparisons would help establish when HMC-based marginalization provides meaningful improvements.

Questions for authors

My questions are based on the weakness above.

  1. Methodological novelty: Could the authors clarify the main methodological novelty of ISO-HMC compared with existing Bayesian Gaussian process approaches that integrate over kernel hyperparameters? In particular, which components are specifically designed for the few-placebo regime, beyond applying standard HMC-based posterior inference?

  2. Theoretical guarantees: Can the authors provide any theoretical justification for the calibration of ISO-HMC credible intervals? For example, under what assumptions on the control sample size N0N_0, covariate dimension dd, and smoothness of the treatment effect function τ(x)\tau(x) should we expect the proposed method to achieve reliable frequentist coverage?

  3. Empirical effectiveness: Could the authors explain when HMC-based hyperparameter marginalization provides meaningful improvements over MAP estimation?

Reviewer_WT93OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript considers uncertainty quantification for the conditional average treatment effect (CATE) when the control arm is much smaller than the treatment arm. The authors proposed ISO-HMC, a structured Gaussian process model that uses Hamiltonian Monte Carlo to account for uncertainty in the treatment-effect kernel hyperparameters and noise variance. However, the current version does not clearly establish the methodological contribution beyond existing Bayesian GP methods, and several claims about calibration and the failure of competing methods are stronger than the analysis and experiments support.

Critical strengths

The manuscript addresses an important problem in CATE inference when only a small control sample is available. Therefore, accounting for hyperparameter uncertainty is well motivated in this setting. The experiments include several synthetic designs and the IHDP benchmark, providing a useful comparison of coverage and interval width across different methods.

Critical weaknesses
  1. ISO-HMC combines a structured Gaussian process model for the conditional average treatment effect (CATE) with HMC-based marginalization over the treatment-effect kernel hyperparameters. However, the manuscript does not clearly illustrate its methodological contribution relative to existing Bayesian GP methods, beyond its application to the few-placebo regime.
  2. The manuscript provides limited justification for the reliability of the proposed GP decomposition. Specifically, the model assumes
Yi=μ0(Xi)+Wiτ(Xi)+εi,μ0∼GP(0,kb),τ∼GP(0,kτ),Y_i=\mu_0(X_i)+W_i\tau(X_i)+\varepsilon_i, \qquad \mu_0\sim GP(0,k_b),\quad \tau\sim GP(0,k_\tau),

When the control arm is small, the estimation of μ0\mu_0 may be highly uncertain, potentially affecting the posterior inference for τ\tau. However, the manuscript does not sufficiently clarify how the specification and uncertainty of the baseline kernel kbk_b affect the resulting CATE intervals, or whether the conclusions are sensitive to alternative kernel and hyperparameter choices. More extensive sensitivity analyses would help establish the reliability of the proposed approach beyond the smooth, low-dimensional settings considered.

  1. The manuscript argues that the unweighted DR-Learner is unbiased but has high variance, whereas inverse-variance weighting may introduce bias. However, it does not clearly distinguish the standard DR score from the weighted regression procedure or specify the conditions under which weighting induces bias.

  2. The explanation of the full X-Learner appears inconsistent with its stated implementation. Section3 states that propensity weighting concentrates on the treated-arm component in the few-placebo regime. However, AppendixB uses τ^(x)=π^(x)τ^0(x)+(1−π^(x))τ^1(x)\hat{\tau}(x)=\hat{\pi}(x)\hat{\tau}_0(x)+(1-\hat{\pi}(x))\hat{\tau}_1(x), where τ^0\hat{\tau}_0 is fitted on control-arm pseudo-outcomes. This equation places more weight on the control-arm component when π^\hat{\pi} is close to one. The authors should correct this argument and clarify which parts of the treated-only bias analysis apply to the full X-Learner.

  3. The experiments do not clearly establish the calibration benefit of HMC over MAP. On the linear design at N0=30N_0=30, Table4 reports coverage of 0.95 for ISO-MAP and 0.99 for ISO-HMC, with interval widths of 0.78 and 0.96, respectively. This shows more conservative intervals, but does not demonstrate that HMC is needed to restore calibration in this setting. In addition, Section5.3 states that ISO-HMC caps the treated sample at 150 observations, while the main synthetic experiments specify N1=500N_1=500. The manuscript should clarify whether ISO-MAP uses the same observations and provide a matched-data comparison to isolate the effect of hyperparameter marginalization.

  4. Although ISO-HMC performs well in smooth, low-dimensional settings, its coverage on the IHDP benchmark falls to 0.76 when the control sample size is 30, substantially below the nominal 95% level. This raises concerns about the reliability of the resulting intervals in more challenging settings.

  5. Several figures and tables are not consistently labeled. Figure1 labels GP-CATE as ``ours,'' although ISO-HMC is the proposed method. Its shared ISO legend refers to ISO-HMC in the synthetic panels but ISO-MAP without the length-scale prior in the IHDP panel. The caption explains this change, but the legend should distinguish the variants directly. Table9 also reports ISO-MAP coverage of 0.467 on IHDP at N0=30N_0=30, compared with 0.75 in Table~4, without identifying a different variant. The authors should reconcile these results and consistently distinguish the models with and without the prior.

Questions for authors

Please see strengths and weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.