- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies treatment allocation in constrained settings, where benefit is defined by the conditional average treatment effect. The authors propose CPO+, which trains the decision model with an SPO+ surrogate to correctly allocate people across groups, using a doubly robust proxy as the true CATE is unobserved. They show the expected surrogate loss upper-bounds causal regret (Thm 3.3). They then define a decision fragility index (DFI) that characterizes the importance of examples along the decision boundary. This motivates DC-CPO, which adds a margin-weighted penalty for examples near the decision-boundary. The paper compares the proposed approaches to simpler regression-based approaches on synthetic and semi-synthetic datasets, and show better decision performance (i.e., lower regret) than the regression baselines.
Critical strengths
- Adapting SPO+ to doubly robust pseudo-outcomes is a natural idea and the authors do a good job of formalizing the problem to lead up to this natural conclusion. I appreciate Figure 1, really driving home how this is amenable to the SPO formulation and motivating the margin discussion.
- Looking at margins along decision-boundary directions is a nice idea that hasn't been explored as well in the treatment effect space. It explains why two estimators with the same error can lead to different allocations a bit more clearly than past work, connecting more to core ML/decision-theory, which comes with nice theoretical proofs.
Critical weaknesses
- The main motivation is not new. Kamran et al. (AISTATS 2024) argue that accurate CATE estimation is sufficient but not necessary for optimal allocation under resource constraints, and that better CATE accuracy does not imply a better allocation. They give a margin argument that is close to much of the theory in this work. The margin argument also lends itself very closely to the learning-to-rank area as a whole. Even more, Rank-Learner (Arno et al. 2026) make the same argument and propose a Neyman-orthogonal pairwise ranking loss for this problem that improves even more on past work Neither paper is cited. The authors should discuss how their contribution differs from this line of work (ranking treatment effects), and discuss whether neyman-orthogonality holds for their methods as well. If not, why should this be preferred over Rank-Learner?
- The experiments are quite limited and and show very limited improvements of the model and show limited results as a whole. Some pointed weaknesses in the experiments:
- The papers in Weakness 1, and ranking methods in general, are missing as baselines. The experiments are directly testing ranking, so even non-CATE methods like LambdaRank would be useful as a baseline.
- The results do not support the claim that accuracy and decision quality diverge. All CPO methods have lower PEHE than MSE-DR and MAE-DR in most settings throughout the paper. The only case of worse PEHE with better regret is DC-CPO vs CPO+ on ACTG-175. This suggests the gains may come from something other than reframing the problem as decision-oriented.
- DC-CPO⁻ performs close to DC-CPO. The nonuniform weighting that Section 4 motivates appears to add little.
- The baselines are linear and may be under-tuned. Specifically, 1) the baselines and the proposed methods are regularized differently (ridge vs no ridge) 2) Propensities clipping is it should be explored a bit more as it could cause the PEHE issues in the bad baseline results.
Questions for authors
- How does the proposed method compare to ranking methods like Rank-Learner? Both in motivation/theory and experimentally.
- Why do the baseline methods do so poorly in terms of PEHE compared to the decision-based estimators?
- Can you better quantify whether the DFI actually flips allocations in real problems/datasets?
- Why consider costs and individual-level baseline reweighting in the problem setting if these are never tested in the experiments?
5 comments
W2 / Q2 — Could unequal regularization, limited tuning, or propensity clipping explain the poor baseline PEHE and the apparent decision gains?
Answer: Wider regularization search with prediction-loss selection lowers MSE-DR PEHE; stronger propensity clipping also lowers its ACTG-175 PEHE. With the strengthened prediction baseline, MSE-DR has lower mean PEHE than DC-CPO on all four datasets, while DC-CPO retains lower mean regret.
1. Regularization and model selection. Appendix C.1 specifies ridge-regularized MSE-DR and MAE-DR, with regularization selected by validation pseudo-regret, while the submitted CPO+ and DC-CPO objectives have no additional parameter ridge. For the supplementary comparison reported here, we use L2 parameter regularization across methods and evaluate prediction-loss-based selection for MSE-DR and MAE-DR over an expanded regularization grid. Data splits, features, and nuisance estimates are shared within the comparison.
2. Strengthened prediction baselines. Out results, 20 paired seeds; each cell reports mean PEHE / mean regret. Original MSE-DR uses the original ridge grid; the baselines selected by prediction loss use that grid augmented with {0.1, 1, 10}.
| DATASET | ORIGINAL MSE-DR (PSEUDO-REGRET) | MSE-DR (PREDICTION LOSS) | MAE-DR (PREDICTION LOSS) | DC-CPO |
|---|---|---|---|---|
| DGP | 0.7408 / 1.3673 | 0.4323 / 1.7892 | 0.4464 / 1.1980 | 0.5952 / 1.0450 |
| Twins | 0.3540 / 0.7822 | 0.3480 / 0.7977 | 0.3950 / 0.9076 | 0.3762 / 0.7418 |
| NHANES | 1.5519 / 1.5161 | 0.8146 / 1.5241 | 1.3489 / 1.4481 | 1.0215 / 1.3414 |
| ACTG-175 | 5.9773 / 1.9371 | 0.9638 / 2.2359 | 1.1643 / 1.8976 | 1.0930 / 1.5642 |
3. Propensity clipping. Increasing the shared propensity clipping threshold from 0.000001 to 0.05 reduces ACTG-175 MSE-DR PEHE from 5.9773 to 4.5026, with smaller changes on the other datasets. The flexible-learner comparisons in Appendix C.6 additionally evaluate forest and Nyström-RBF representations.
Appendix C.7 now includes the comparison of regularization search ranges and validation criteria.
Q3 — Can DFI be linked quantitatively to actual allocation changes on the datasets?
Answer: DFI measures how much perturbation is needed to reach an allocation boundary. Its predicted-center form is positively associated with bootstrap allocation changes on all four datasets.
1. Critical perturbation radius. For a unique reference optimum and perturbations in , the critical radius is . Perturbations inside this radius preserve the reference allocation; a competing allocation can tie at the boundary, and moving beyond it along a most sensitive direction changes the allocation. Thus a larger DFI corresponds to a smaller perturbation needed to reach a decision boundary.
2. Allocation changes and empirical association. Across 20,000 center–batch combinations on four datasets, using both predicted and oracle centers, scaling the constructed critical perturbation by 0.99 preserved the allocation in every case, while scaling it by 1.01 changed the allocation in every case. The table reports the empirical association between DFI and bootstrap switching on the synthetic and real-covariate semi-synthetic benchmarks.
| DATASET | MEDIAN CRITICAL RADIUS¹ | BOOTSTRAP SWITCH RATE | PREDICTED-CENTER SPEARMAN [95% CI] | ORACLE-CENTER SPEARMAN [95% CI] |
|---|---|---|---|---|
| DGP | 0.1542 | 85.45% | 0.439 [0.405, 0.472] | 0.054 [0.016, 0.092] |
| Twins | 0.2367 | 73.28% | 0.556 [0.522, 0.589] | 0.077 [0.036, 0.120] |
| NHANES | 0.1868 | 79.68% | 0.513 [0.480, 0.547] | 0.060 [0.026, 0.095] |
| ACTG-175 | 0.1736 | 81.75% | 0.479 [0.442, 0.515] | 0.068 [0.032, 0.103] |
¹ The critical-radius median uses the predicted center. We use CPO+ Out predictions from 50 seeds, 50 batches per seed, and 40 saved bootstrap refits; is estimated from the bootstrap benefit predictions. A switch denotes any change in the batch allocation. Spearman correlations are computed across batches within each seed and then averaged; confidence intervals resample seeds. DFI and switching frequencies use the same bootstrap draws.
Appendix C.7 now includes the critical-radius explanation, perturbation checks, and empirical association table.
W2 — DC-CPO⁻ performs close to DC-CPO. What does nonuniform direction weighting contribute?
Answer: Nonuniform weighting reduces Crossing by 0.63–1.07 percentage points and reduces regret relative to DC-CPO⁻ on all four datasets.
1. What the ablation retains. DC-CPO⁻ retains active-boundary selection, directional projections, and the boundary penalty, with uniform direction weights. Its comparison with DC-CPO therefore measures the additional contribution of how the penalty is distributed across directions.
2. Paired comparison. Out ablation, 50 paired seeds. Method columns report mean regret / mean Crossing, where Crossing is the whole-allocation bootstrap self-switch probability. Gains are DC-CPO⁻ minus DC-CPO; positive values favor DC-CPO. Crossing gains are in percentage points (pp), and intervals are paired-seed bootstrap 95% confidence intervals.
| DATASET | CPO+ | DC-CPO⁻ | DC-CPO | REGRET GAIN [95% CI] | CROSSING GAIN, PP [95% CI] |
|---|---|---|---|---|---|
| DGP | 1.1121 / 0.8544 | 1.1001 / 0.8495 | 1.0862 / 0.8431 | 0.0139 [0.0035, 0.0235] | 0.64 [0.50, 0.78] |
| Twins | 0.7207 / 0.7329 | 0.7033 / 0.7286 | 0.6839 / 0.7207 | 0.0194 [0.0020, 0.0204] | 0.79 [0.56, 1.04] |
| NHANES | 1.3880 / 0.7968 | 1.3152 / 0.7724 | 1.2969 / 0.7661 | 0.0183 [0.0062, 0.0304] | 0.63 [0.47, 0.79] |
| ACTG-175 | 1.6386 / 0.8175 | 1.5460 / 0.7986 | 1.5169 / 0.7879 | 0.0292 [0.0141, 0.0441] | 1.07 [0.84, 1.30] |
Gains use the paired seed-level estimates; displayed method means are rounded.
All reported paired intervals are above zero.
Appendix C.7 now includes this paired ablation, and Section 4 explains the incremental role of direction weighting.
Q4 — Costs and individual benefit weights appear in the formulation. How are they tested in the experiments?
Answer: The original experiments already use heterogeneous costs with equal benefit weights; the supplementary comparisons test zero costs and individual benefit weights. DC-CPO has lower regret than CPO+ under the tested individual weights on all four datasets.
1. Utility specification and experimental setup. Known costs and individual weights translate CATE into decision utility through . The main setting uses , , and ; allocation-independent baseline terms are constants. The added comparisons use zero costs and , applying each utility consistently in training, validation, and evaluation.
Section 6 now clarifies the utility specification, and Appendix C.7 includes the comparison.
2. Experimental results. Out regret gain = CPO+ − DC-CPO [95% CI], 20 paired seeds; positive values favor DC-CPO. Compare methods within each column's utility.
| DATASET | ZERO COST, EQUAL WEIGHTS | HETEROGENEOUS COSTS, EQUAL WEIGHTS | HETEROGENEOUS COSTS + INDIVIDUAL REWEIGHTING |
|---|---|---|---|
| DGP | 0.0447 [0.0090, 0.0585] | 0.0052 [0.0021, 0.0536] | 0.2679 [0.1635, 0.3804] |
| Twins | 0.0191 [0.0143, 0.0548] | 0.0066 [0.0015, 0.0570] | 0.0989 [0.0212, 0.1819] |
| NHANES | 0.0227 [0.0128, 0.0854] | 0.0785 [0.0331, 0.1231] | 0.1680 [0.0079, 0.3321] |
| ACTG-175 | 0.1245 [0.0141, 0.2427] | 0.1130 [0.0302, 0.1955] | 0.5478 [0.3461, 0.7533] |
We would welcome your feedback on whether we have addressed your concerns, and we would be happy to continue the discussion and provide any further clarification to support your re-evaluation.
We have added ranking and stronger-baseline comparisons, DFI diagnostics, and utility tests.
Answer: DC-CPO regularizes errors along contrasts between feasible allocations; CPO+ and DC-CPO generally retain first-order nuisance sensitivity after the nonlinear allocation map. In the reported comparison, DC-CPO has lower mean Out regret than both Rank-Learner validation variants on all four datasets.
1. Relation to treatment-effect ranking. Kamran et al. (2024) establish that accurate CATE estimation is sufficient but unnecessary for good allocation and develop a margin argument; Arno et al. (2026) turn the ranking perspective into a Neyman-orthogonal pairwise loss.
2. Allocation geometry and statistical properties. For our partition-capacity experiments, net-benefit ordering and sign determine an optimizer under consistent tie handling, making ranking a direct comparator. Under more general constraints, a direction can compare two feasible allocations through sums of individual net benefits. DFI measures directional uncertainty relative to the corresponding allocation margin; Theorems 4.1 and 4.3 translate this quantity into an invariance certificate and a crossing bound. DC-CPO uses nearby allocation contrasts to assign the penalty to decision-sensitive errors.
The AIPW guarantee concerns the pseudo-benefit's conditional mean. We evaluate the practical value of DC-CPO's allocation-geometry approach against pairwise and Rank-Learner adaptations using the same costs, constraints, and downstream regret.
3. Experimental comparison. Out regret; mean (SE), 30 matched seeds. Rank-Learner is the linear-feature plus DR-calibration adaptation; the two labels denote validation criteria.
DC-CPO also has lower mean Out regret than CPO+ and DR-Pairwise on all four datasets. The comparison uses the same downstream allocation criterion for all methods.
Appendix C.7 now includes the ranking comparison and its adaptation and validation procedures, while Appendix A presents the motivation and theory comparison.