- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper considers budget-constrained allocation based on estimated CATEs relative to selecting a policy through empirical welfare maximization (EWM) directly. Under a correctly specified shared treatment effect model, inverse-variance weighted CATE estimation is shown to achieve lower local allocation regret, as it uses information across population groups to learn a common parameter. For unweighted CATE, these gains also depend on the noise in the pooled observations. On the other hand, misspecification can reverse these gains.
Critical strengths
- Interesting question: is CATE-based allocation preferable to EWM?
- Interesting answers, with precise statements about local regret and crossover points.
- Informative simulations to study these theoretical results.
- Extends the comparison to cross-fitted doubly robust scores under nuisance-estimation conditions.
- Interesting theory results, in particular that observations that cancel out of the empirical welfare difference can still be useful to learn the shared treatment effect parameter. The resulting thresholds make the tradeoff interpretable.
Critical weaknesses
My main concern is that it is not always clear that fixing the policy class means both procedures use the same restrictions on the treatment effect function. Particularly, the main allocation comparison is between treating one higher-cost individual and treating two lower-cost individuals together. Under correct specification, the CATE estimator treats the effects for the two lower-cost groups as fixed, while EWM estimates those effects with noisy data. So the results seem to indicate that the CATE estimator can outperform not only because it is pooling data across groups but also because it is using correct additional restrictions about treatment effects. Appendix L acknowledges that the comparison is between specific procedures, but this still seems important for interpreting the source of the advantage. I would be interested in learning how the results would change if these effects and the relation across groups were also estimated.
More generally, the theory results and simulation studies are in very stylized settings: four observed population groups, one unknown parameter, deployment batches of three, and two competing policies that differ only on one batch composition, with performance compared in a local neighborhood of the allocation boundary. So it is unclear how general these insights are for budget-constrained allocation questions more broadly. Even the extension to binary outcomes maintains this basic treatment effect structure. Related to my first point above, the misspecification analysis only changes the treatment effect for an additional group whose observations contribute to the shared CATE fit but cancel out of the empirical welfare difference. While this does neatly show cases where pooling information is bad, it also leaves the EWM comparison unchanged in the theoretical analysis. It is unclear how relevant this is relative to other forms of misspecification.
Questions for authors
-
Would it be useful to consider a third estimator that learns from observations of the higher-cost group alone, while retaining the assumed fixed effects for the two competing lower-cost groups? This could help tease apart the benefits of pooling data across groups from those of using the other restrictions in the CATE model.
-
Are there any insights you can share about how the results might change in more general settings—for instance, if the effects currently treated as fixed and the relation across groups had to be estimated, or if there were more allocation boundaries?
-
Could the misspecification theory results be extended to changes in treatment effects for the groups directly involved in the allocation comparison? Which insights about the reversal in local regret ordering would still hold more generally?
7 comments
Response to Reviewer ayJ4's comments (2/5)
...continued from above
3. Learning the sharing relation requires additional identification conditions in this local regime. This addresses the further question in W1 about estimating the relation across groups. The preceding extension estimates the low-cost effects while retaining known baselines and loadings. Estimating an unrestricted loading is a different problem because the information about that loading need not increase along the local sequence.
For example, even if were known, the Gaussian score experiment provides expected Fisher information
=\frac{np_N\theta_n^2}{\sigma_N^2} =\frac{p_Nh^2}{\sigma_N^2}=O(1).$$ For two sufficiently close fixed loadings, the Kullback–Leibler divergence is $p_Nh^2(\kappa_1-\kappa_0)^2/(2\sigma_N^2)$, so they cannot be distinguished with vanishing error. Hence, in this Gaussian local experiment, no estimator is consistent at every loading in an open interval. External calibration or additional identifying restrictions require separate analysis. This does not rule out useful pooling when, for example, the loading’s sign or range is known. It means that the known-loading efficiency formula cannot automatically be retained. The E-only comparison separates the benefit of known low-cost effects from the additional gain from pooling. **The pooling gain persists when the low-cost effects are estimated under the correct, known sharing relation**, as shown by the variance comparison and Table A1. Learning the relation itself requires separate analysis: information about an unrestricted loading remains bounded in this local experiment, so the known-loading constants cannot automatically be retained. As discussed in Appendix L, these conclusions concern the specified procedures and assumptions and do not establish a universal advantage of CATE-based allocation over EWM. **What we revised:** In the revised manuscript, Appendix M.6 now includes the E-only comparison, the variance derivation with estimated low-cost effects, and the oracle-score and cross-fitted results (Table 16). Estimating the sharing relation under additional identification conditions remains future work.Response to Reviewer ayJ4's comments (3/5)
W2&Q3: "Scope of the comparison and misspecification beyond the auxiliary group"
Thank you for these related questions. The additional analyses extend the allocation settings and show that reversal can persist beyond the original -only deviation under the stated local conditions. Specifically, we vary the batch size, budget, and candidate policy class to assess sensitivity to the allocation setting. We then use local derivations and corresponding experiments to examine reversal when decision-critical effects also change.
The advantage persists in the tested larger batches and policy class. To examine the allocation structure, we use larger batches and different budgets, while retaining costs . In the Gaussian local experiment, with , , and 50,000 training replications per sign, weighted CATE has lower mean regret in all five configurations and both signal directions. The competing policies differ on 1, 6, 21, 25, and 19 batch compositions, respectively. These configurations retain the same local bundle contrast. Our supplementary derivation establishes that their different composition probabilities change the common regret multiplier, preserving the ordering under the specified conditions. Thus, this extension relaxes the three-person batch and single-composition restrictions while retaining one active boundary.
Separately, a Bernoulli experiment compares the criteria over 11 distinct policies induced by the full one-dimensional shared score class. Both procedures use the same score class and decoder, with EWM maximizing complete-U welfare over the induced policy class. S-learner and T-learner outcome regressions are used to construct cross-fitted doubly robust scores.
Here the enlarged class uses , , and . The 11 policies span this full parameter range. Treated response probabilities are clipped to , and the actual CATE is . No clipping was recorded in the simulated samples. This sample-level observation does not by itself establish exact sharing throughout the population. The results are interpreted as empirical evidence for this specified design.
S-learner results are shown for all deployment configurations and both parameter signs. Each sign uses and five seeds with 20,000 training replications each, totaling 100,000. Regret is evaluated on independent deployment batches against the optimal budget-feasible allocation under the actual CATE .
Table A2. Mean per-person allocation regret multiplied by in the 11-policy experiment.
| Weighted CATE, | EWM, | Weighted CATE, | EWM, | ||
|---|---|---|---|---|---|
| 3 | 2 | 48.6752 | 123.5081 | 47.7641 | 101.6427 |
| 5 | 2 | 23.6024 | 106.4002 | 42.9180 | 142.7502 |
| 8 | 2 | 12.3648 | 90.5929 | 33.8971 | 119.1577 |
| 8 | 4 | 24.9020 | 126.6499 | 49.3034 | 176.5532 |
| 8 | 6 | 37.8540 | 144.0882 | 58.6636 | 129.8624 |
Weighted CATE has lower mean regret in all ten S-learner comparisons in Table A2, and the corresponding T-learner experiments give the same ordering. These results show that the observed advantage is not confined to a two-policy search. They remain a structured, nonlocal numerical comparison and do not establish the original regret constants or reversal threshold for multiple active boundaries. We also relax the single-parameter restriction by estimating the two low-cost effects. In the Gaussian experiment with estimated scores and weights, the resulting error probability is , compared with for EWM, indicating a smaller but remaining pooling advantage.
Decision-critical perturbations can produce reversal even when the true allocation contrast changes. To determine when reversal persists beyond the original -only path, retain the original baseline, known loadings, positive type probabilities and conditional score variances, and consider any fixed vector
\qquad \tau_n=b+\frac{d}{\sqrt n}.$$ Assume the other allocation comparisons remain separated, so the oracle remains in the same two-policy class. Define $$K=d_E-d_1-d_2,\qquad M=\frac{p_Er^2}{\sigma_E^2} +\frac{p_N\kappa^2}{\sigma_N^2},$$ $$a(d)=\frac{p_E r d_E}{\sigma_E^2} +\frac{p_N\kappa d_N}{\sigma_N^2}, \qquad T_D=\frac{\sigma_E^2}{p_E} +\frac{\sigma_1^2}{p_1} +\frac{\sigma_2^2}{p_2}.$$ Here $p_t$ and $\sigma_t^2$ are the type probabilities and conditional oracle-score variances. ...
Response to Reviewer ayJ4's comments (1/5)
We thank you for your review and suggestions. Our responses are as follows:
W1&Q1: "Sources of the CATE advantage"Known low-cost effects contribute to the original advantage, but they do not fully explain it: pooling still improves local regret when these effects are estimated under the correct sharing relation. To separate these two sources of improvement, we introduce an E-only comparator and then allow the low-cost effects to be estimated. We also address the further question of learning the sharing relation itself.
1. The E-only comparison separates the benefit of known effects from the additional benefit of pooling. We consider
\operatorname{clip}_{[-\bar\theta,\bar\theta]} \{\frac{\bar\Gamma_E-b(E)}{r}\},$$ which retains the known low-cost effects and the original decoder while excluding $N$ from parameter estimation. Under the correctly specified sequence $\theta_n=sh/\sqrt n$, its local index is $I_E=h\sqrt{H_E}$, giving $$\sqrt n \mathbb E R_3(\widehat\pi_E) \longrightarrow 2p_Ep_1p_2hr \Phi(-I_E).$$ Using the notation of Eqs. (28) and (42), Eq. (38) decomposes as $$I_D^2=\frac{h^2r^2}{T_D} <I_E^2=h^2H_E <(I_R^*)^2=h^2(H_E+H_N).$$ The first improvement uses the known low-cost effects. The second isolates the additional information from $N$, with the same known effects and decoder. **2. Under correct specification, pooling still lowers local regret when the low-cost effects are estimated.** We also fit $$f_{\theta,\gamma_1,\gamma_2} =b+(r\theta,\gamma_1,\gamma_2,\kappa\theta)$$ using inverse variance weighting (IVW) in the squared loss. Both criteria use this enlarged class within a fixed parameter neighborhood preserving the same induced policies. The simulation uses $[-0.1,0.1]^3$. Under the correctly specified sequence $(\theta_n,\gamma_{1,n},\gamma_{2,n})=(sh/\sqrt n,0,0)$, the centered fitted bundle contrast multiplied by $\sqrt n$ has limiting variance $$V_{\mathrm{pool}} =\frac{r^2}{H_E+H_N} +\frac{\sigma_1^2}{p_1} +\frac{\sigma_2^2}{p_2} <T_D.$$ Its limiting wrong-allocation probability is therefore $\Phi(-hr/\sqrt{V_{\mathrm{pool}}})$, showing that a pooling benefit remains when the low-cost effects are estimated. This oracle-score experiment uses $n=2500$, $p=(0.2,0.2,0.2,0.4)$, $r=\kappa=h=1$, and unit conditional score variances. Entries are means across five seeds, with across-seed standard deviations in parentheses. Each seed uses 100,000 replications. All methods use the same simulated observations. Differences below are computed from unrounded estimates. **Table A1. Wrong-allocation probabilities under correct specification.** | Estimator | $L_1/L_2$ effects | Simulated probability (SD) | Local limit | |---|---|---:|---:| | EWM, raw complete-U criterion | Empirical score means | 0.3976 (0.0011) | 0.3981 | | E-only | Known | 0.3277 (0.0016) | 0.3274 | | weighted CATE | Known | 0.2191 (0.0009) | 0.2193 | | weighted CATE estimating $L_1/L_2$ | Estimated | 0.3842 (0.0008) | 0.3848 | Expected allocation regret equals each probability in Table A1 multiplied by the common value gap, $3.2\times10^{-4}$. Compared with EWM, E-only reduces error by approximately 6.9960 percentage points using the known low-cost effects. CATE estimation with inverse variance weighting further reduces error by 10.8602 percentage points relative to E-only through pooling. With estimated low-cost effects, the improvement over EWM is smaller but remains positive: 1.3436 percentage points, with a paired 95\% Monte Carlo interval of [1.2362, 1.4510] percentage points. ...