- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CUBO combines Bayesian-bootstrap population weights with a fitted nuisance posterior and learns a treatment policy using posterior mean welfare plus lower-tail CVaR. The paper derives conditional simulation and welfare-regret bounds and evaluates several Bayesian neural backbones on IHDP and ACIC policy-learning tasks.
Critical strengths
-
A coherent decision-level uncertainty objective. Common posterior scenarios and whole-function nuisance draws preserve relevant dependence across policies; posterior CVaR targets aggregate-welfare uncertainty rather than individual outcome risk.
-
Useful theoretical care. The analysis covers the actual mean–CVaR decision and finite draw bank, retains optimization error explicitly, and distinguishes cross-fitted from full-sample nuisance estimation.
Critical weaknesses
-
The central objective is not isolated by the ablations. Table 1 changes nuisance fitting and policy optimization together, without a posterior-mean plug-in or a learner using the same Bayesian nuisance model. Figure 4 starts at , while bootstrap-only and nuisance-only variants are also absent; the gains therefore do not establish whether the proposed tail objective and combination of uncertainty sources add value beyond the changed estimator.
-
The evaluation does not directly test the advertised downside protection. Section 5 reports average normalized regret and policy accuracy, which are useful outcome measures but do not reveal whether posterior tail scenarios correspond to genuinely low-welfare decisions. The limited outcome-generating settings leave this question unresolved; a focused repeated-dataset check within an existing benchmark would be more informative about the central claim than further optimization runs.
-
The combined welfare posterior rests on an unvalidated modeling approximation. Equation (7) assumes conditional independence between population and nuisance uncertainty, although both are learned from the same observations. The paper explicitly acknowledges this approximation, so it is not a hidden error, but does not establish its effect on adverse welfare scenarios; this matters because CVaR makes those scenarios directly determine the learned policy.
-
The experiments do not directly instantiate the main theoretical construction. Sections 3.3–4 formulate cross-fitted nuisance scores, whereas the reported experiments use full-sample fitted effects. A separate full-sample extension is provided, but its additional complexity and nuisance-bias conditions are not checked for these learners; I therefore cannot treat the theoretical guarantees as established explanations for the observed experimental gains.
-
The comparison with confidence-based policy learning mixes objectives and policy classes. Section 3.1 contrasts posterior CVaR with a simultaneous lower-confidence-bound objective, but Table 1 implements PoLeCe over a finite grouped-assignment class while CUBO learns a neural policy. This is a legitimate comparison of complete procedures, yet cannot determine whether the Bayesian tail criterion itself provides the claimed advantage over confidence-based decision making.
Questions for authors
-
Using the existing fitted nuisance draws and the same policy class and budget, how do CUBO, , the posterior-mean plug-in, and the two single-source uncertainty variants compare?
-
Within an existing benchmark's outcome-generating setting, does a focused repeated-dataset check show that optimizing the posterior lower tail reduces actual low-welfare outcomes? What evidence supports the combined posterior's representation of those scenarios?
-
Which full-sample theoretical conditions are intended to cover the reported learners, and how should Table 1's PoLeCe comparison be interpreted given the different policy classes?
2 comments
R3. Is conditional independence between population and nuisance uncertainty validated?
Thanks for your good question. Conditional independence defines the scenario law in Equation (7):
We added concordant and discordant whole-draw pairings that preserve the saved population-weight and nuisance marginals while changing their dependence. For noisy IHDP with Bayesian ESCFR, mean upper-5% regret is 1.797 under concordant pairing and 1.461 under discordant pairing, compared with 1.666 under independent pairing. Their purpose is to quantify how coupling can change the decision. For welfare laws and , the hinge representation gives
The factor makes dependence sensitivity particularly relevant at small . We have stated the conditional-independence approximation in Section 2.3.
R4. Do the experiments instantiate the cross-fitted theoretical construction?
The experiments use full-sample fitted effects. The cross-fitted orthogonal result and Proposition 2 cover different constructions. Cross-fitting permits conditional empirical-process control for held-out scores. For the experimental score , Proposition 2 instead includes joint policy–nuisance complexity:
\qquad \mathcal F=\{\phi_\theta(\cdot;\pi):\pi\in\Pi,\theta\in\Theta\}.$$ Its nuisance bias is first order, i.e., $b_n\le\int\|\tau_\theta-\tau_0\|_{L_2(\mathcal{P}_0)}q_n(d\theta).$ Freezing a model before policy optimization fixes the conditional simulation problem, while its statistical dependence on the fitting sample remains part of the full-sample analysis. We have made the score distinction explicit in Section 3.3 and Appendix D.3 and clarified the full-sample guarantee's scope in Section 4 and Appendix C.4. ### R5. How should PoLeCe be interpreted when its policy class differs? **The main-paper comparison evaluates complete procedures. We added a separate common-class experiment to isolate the selection criterion.** All four criteria receive the same eight candidate policies, observed-validation posterior DR scores, nuisance information, and observations. CUBO uses $\beta=.5$, $\alpha=.10$, and $\lambda=.75$. The simultaneous LCB uses its internal 95% confidence calibration and 2,000 multiplier draws. The following BDeRCFR results are means over 200 realizations: | Criterion | IHDP upper 5% regret | IHDP upper 20% regret | ACIC upper 5% regret | ACIC upper 20% regret | |---|---|---|---|---| | Posterior mean | 1.934 | 0.657 | 0.513 | 0.224 | | Mean–CVaR | 1.953 | 0.670 | 0.620 | 0.294 | | CUBO | **1.823** | **0.595** | **0.423** | **0.173** | | Simultaneous LCB | 1.977 | 0.682 | 0.661 | 0.309 | The table shows that CUBO achieves the lowest upper-5% and upper-20% regret on both datasets within the same candidate-policy class. Bold values are best within each metric column. Thus these results further support CUBO's contribution to reducing upside regret under shared nuisance information. Because validation also selected the nuisance checkpoint, the comparison evaluates criterion behavior under shared information. Appendix D.2 describes the PoLeCe baseline, and Appendix D.6, Table 5 reports this comparison.
Thank you for recognizing the value of our decision-level uncertainty objective! We also greatly appreciate your careful reading and constructive feedback, and we are more than glad to provide our responses below.
R1. Can the objective and uncertainty sources be isolated using the existing nuisance draws?
Yes. The added controls reuse the saved nuisance draws and keep the candidate policies fixed. With τˉ(Xi)=S−1∑sτθs(Xi), the four scenario contributions are:
Removing CVaR sets λ=0 while retaining the mean component. The following comparisons use the same fitted model, eight candidate rules, 100-draw bank, and fixed loss settings, with β=.5, α=.10, and λ=.75. Entries are test means over 200 realizations in outcome units. Bold full-method values are lower than every displayed ablation in the same row.
Full CUBO achieves the lowest upper-5% and upper-20% regret in every reported matched case. Thus the saved-draw controls demonstrate gains from CVaR and nuisance variation, with smaller gains from bootstrap reweighting. For the positive mean–CVaR reward alone, fixed effects and unrestricted treatment make the sign plug-in optimal in every bootstrap-only scenario. Appendix D.6, Table 4 reports these matched comparisons.
R2. Do repeated datasets show actual downside protection?
Thanks for the concern. Following your suggestion, we have added results evaluating individual-regret tails across repeated realizations in datasets. For a test split of size m, the paper's notation gives τi=μ1,0(Xi)−μ0,0(Xi), πi=π(Xi), and ri=max(τi,0)−πiτi. We evaluate $ \rm{Reg}(\pi)=\frac1m\sum_i r_i,$ and
\quad \rho\in\\{0.05,0.20\\}.$$ We average these metrics equally over 200 distinct realizations per dataset. Each Bayesian plug-in is followed by CUBO using the same fitted model and candidate rules. Bold CUBO entries improve over the plug-in method. **Noisy IHDP and ACIC 2017 high-noise over 200 realizations.** | Method | IHDP upper 5% regret | IHDP upper 20% regret | ACIC 2017 upper 5% regret | ACIC 2017 upper 20% regret | |---|---|---|---|---| | T-Learner | 4.384 | 1.953 | 0.923 | 0.422 | | BT-Learner plug-in | 3.207 | 1.343 | 0.739 | 0.312 | | BT-Learner + CUBO | **2.765** | **1.051** | **0.510** | **0.196** | | CFRNet | 4.614 | 2.150 | 1.085 | 0.731 | | BCFRNet plug-in | 1.382 | 0.421 | 1.064 | 0.734 | | BCFRNet + CUBO | 1.382 | 0.421 | **0.956** | **0.395** | | TARNet | 4.896 | 2.117 | 1.133 | 0.535 | | BTARNet plug-in | 3.000 | 1.094 | 0.765 | 0.328 | | BTARNet + CUBO | **2.896** | **1.034** | **0.590** | **0.218** | | Dragonnet | 4.895 | 2.124 | 1.092 | 0.473 | | BDragonnet plug-in | 2.987 | 1.124 | 0.826 | 0.337 | | BDragonnet + CUBO | **2.806** | **1.001** | **0.689** | **0.247** | | DeRCFR | 2.029 | 0.709 | 0.525 | 0.249 | | BDeRCFR plug-in | 1.620 | 0.507 | 0.435 | 0.224 | | BDeRCFR + CUBO | **1.588** | **0.480** | **0.403** | **0.182** | | ESCFR | 1.685 | 0.563 | 0.995 | 0.682 | | Bayesian ESCFR plug-in | 1.389 | 0.429 | 0.970 | 0.671 | | Bayesian ESCFR + CUBO | **1.361** | **0.415** | **0.963** | **0.600** | | CEVAE | 2.870 | 0.952 | 1.463 | 0.923 | | PT-CATE | 5.828 | 3.333 | 1.496 | 0.805 | | DR-EWM | 6.752 | 4.168 | 1.116 | 0.441 | | PoLeCe | 6.732 | 4.046 | 0.882 | 0.420 | CUBO reduces both regret tails for five of six Bayesian backbones on noisy IHDP and all six on ACIC 2017. For example, BT-Learner upper-5% regret on IHDP decreases from 3.207 to 2.765. These reductions indicate lower severity of costly treatment errors on both datasets. Across the two noisy benchmarks, adding CUBO lowers both regret tails in almost all Bayesian comparisons. These results support its effectiveness in reducing policy regret across repeated datasets. The revised paper reports the noisy regret tails in Appendix D.5, Tables 2–3, and distinguishes them from posterior welfare CVaR.