acceptodds
Under review as a conference paper at ICLR 2027

Learning Treatment Effects during Resource Allocation via Priority-Queue Randomization

Abstract

Public service programs often allocate limited resources under uncertainty about their benefits, creating a need for randomization to support credible evaluation. In practice, however, applicants commonly enter waitlists where resources are prioritized toward individuals judged to have higher need through tiered priority queues, making direct randomization difficult. Motivated by this, we develop an experimental design framework for learning treatment effects while treating those most in need where incoming applicants are randomized into priority queues based on their assessed risk scores. Treatments are then provided across queues in priority order and first-in-first-out within queue as budget becomes available. Our contributions are two-fold. First, we characterize what causal effects are identified under this priority-queue allocation. When arrivals are exogenous, treatments are conditionally randomized, and hence standard estimands are identified; when arrivals are endogenous, queue randomization instead provides an instrument for treatment, identifying local treatment effects induced by the queuing process. Second, we develop optimized queue-assignment designs that trade off statistical efficiency against prioritizing higher-need applicants. We show in the process that, despite dependence in treatment assignments induced by the design, usual iid efficiency bounds remain well-justified design objectives. We illustrate the proposed designs using data from a housing allocation program in a large U.S. county.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Causal inference, experimental design, resource allocation, public services

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YczWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a new randomized assignment procedure for estimating treatment effects under resource constraints. It derives causal identification results under exogenous and endogenous arrivals and optimizes queue assignments to balance statistical efficiency and allocation utility.

Critical strengths

The motivation is interesting and practically relevant. The theoretical treatment of dependent allocations and the integration of causal inference with constrained experimental design are potentially valuable.

Critical weaknesses
  1. My main concern lies in insufficient design justification. The paper does not convincingly explain why randomizing priority-queue assignments is preferable to alternative designs, such as stratified lotteries or randomization within priority tiers, under comparable operational and fairness constraints.

  2. Also, comparisons are primarily against two heuristic queue-assignment methods and an RCT baseline. Since the proposed method explicitly optimizes the evaluation objective, outperforming these heuristics provides limited evidence of a substantive methodological advantage.

  3. The framework assumes outcomes are unaffected by treatment timing or queue placement, which may be unrealistic in public service settings where waiting times can directly influence outcomes.

  4. I think theory and experiments are somewhat disconnected as well. The experiments prescribe queue-specific treatment probabilities without sufficiently demonstrating that these correspond to the dynamic priority-queue mechanism analyzed theoretically. Moreover, the empirical analysis relies heavily on semi-synthetic outcomes with homogeneous treatment effects. Under endogenous arrivals, the IV estimand can depend on the design itself, making comparisons under treatment effect heterogeneity less straightforward.

Questions for authors

See my comments in the weakness section

4 comments

Dear Reviewer YczW,

Thank you for your feedback! We address your concerns below:

My main concern lies in insufficient design justification. The paper does not convincingly explain why randomizing priority-queue assignments is preferable to alternative designs, such as stratified lotteries or randomization within priority tiers, under comparable operational and fairness constraints.

Thank you for giving us this opportunity to elaborate more on the alternatives! As you point out, one natural alternative is to assign applicants deterministically to priority tiers using, for instance, cutoffs on estimated risk scores, and then randomize within each tier. Such designs, however, can raise ethical concerns in a waitlist setting, as we discuss in our paper. If randomization determines in advance that an applicant will not receive treatment during the allocation horizon, we believe that decision should be communicated promptly. Keeping that person on an apparently active waitlist despite their predetermined exclusion can delay their pursuit of other options and impose avoidable waiting costs.

Repeated lotteries that maintain everyone’s eligibility could address the concern above. This approach, however, replaces FIFO within each queue and therefore allows later arrivals to receive treatment ahead of applicants who have waited longer, changing an important procedural-fairness feature of systems where waiting time establishes priority.

We therefore view our design as a practical middle ground that addresses these concerns and respects existing operational systems and constraints, while introducing randomness that allows program administrators to learn treatment effects. We will augment the Introduction to explain these alternatives and their ethical and operational considerations more explicitly!

Also, comparisons are primarily against two heuristic queue-assignment methods and an RCT baseline. Since the proposed method explicitly optimizes the evaluation objective, outperforming these heuristics provides limited evidence of a substantive methodological advantage.

We appreciate the constructive criticism! We however would like to emphasize that, to our knowledge, no existing experimental design method jointly addresses the trade-off between prioritization and randomization, sequential arrivals, and resource availability. Wilder and Welle (2025), the work we build off of, consider a static setting without these sequential components. Establishing a principled design approach under these constraints is a central methodological contribution of our work.

In particular, our contribution is not simply applying an optimizer to an already established objective. As can be seen, we characterize how queue randomization induces treatment probabilities, justify an efficiency criterion despite the resulting dependent treatment assignments, and obtain a tractable constrained optimization problem. Setting the ethical concerns aside, the alternative allocation mechanisms mentioned above would require additional analysis to yield a principled experimental design. They are therefore not directly available as comparable existing methods, which motivates our comparison with natural heuristics that practitioners might otherwise use.

The framework assumes outcomes are unaffected by treatment timing or queue placement, which may be unrealistic in public service settings where waiting times can directly influence outcomes.

Thank you for raising this important question! We agree that waiting times can plausibly affect outcomes. As we explain in our response to Reviewer BrDM, when the goal is to capture dependence of outcomes on time to treatment, the appropriate treatment variable is the treatment time. After redefining treatment appropriately, queue assignment can remain a valid instrument, provided it affects outcomes only through this expanded treatment. Formally characterizing and optimizing such an estimand, however, would require additional assumptions and IV theory for multivalued or continuous treatments that go beyond the scope of our work. Further, program evaluations in public-policy settings like those motivating our work often model treatment as binary; we therefore believe our design and analysis tools are still valuable when this approximation is appropriate.

We nevertheless compare treatment times under our optimized design and strict prioritization as a descriptive check. Among applicants treated under both policies, the maximum waiting-time CDF discrepancy across risk-score deciles is 5.33%, which is modest. Please refer to our response to Reviewer BrDM for details; we will add explicit discussion of this limitation.

I think theory and experiments are somewhat disconnected as well. The experiments prescribe queue-specific treatment probabilities without sufficiently demonstrating that these correspond to the dynamic priority-queue mechanism analyzed theoretically. Moreover, the empirical analysis relies heavily on semi-synthetic outcomes with homogeneous treatment effects. Under endogenous arrivals, the IV estimand can depend on the design itself, making comparisons under treatment effect heterogeneity less straightforward.

Thank you for raising these concerns, and our apologies for the confusion! As we explain in our response to Reviewer LT6P, Theorem 4 characterizes the limiting design-induced treatment probabilities, whereas Figure 1 evaluates efficiency–utility trade-offs under prescribed queue-specific treatment rates. In finite samples, variation in queue composition and the relative timing of arrivals and treatment releases can allow some lower-priority applicants to receive treatment while some higher-priority applicants remain untreated. Thus, the finite-sample treatment probabilities need not equal their limits.

To demonstrate the connection to the dynamic allocation mechanism, we conducted a new simulation of the full arrival and priority-queue allocation process, building on the endogenous-arrival setting in Figure 2. We used a treatment budget of 50% and a design targeting 98% of maximum anticipated utility. The budget release process satisfies Assumption 2 asymptotically and yields finite-sample queue treatment probabilities of approximately (0.9, 0.1). Across 1,000 repetitions at sample sizes of 500, 1,000, and 2,000, the IV estimator exhibits negligible bias and coverage close to the nominal 95% level with both oracle and estimated nuisances. Please refer to our first response to Reviewer BrDM for details. We will incorporate this experiment and distinguish the planning rates from the limiting probabilities more explicitly.

Regarding heterogeneous effects, as you mention and as Theorem 2 shows, the IV estimand generally depends on the queue-assignment policy under unrestricted heterogeneity. Consequently, some kind of restriction is necessary to obtain an estimand invariant to designs. We believe the homogeneous PLIV model, despite imposing a strong additive homogeneity restriction, offers a natural starting point.

A useful extension is to only restrict arrival-relevant heterogeneity through a ``no interaction'' assumption such as E(Y1−Y0∣X,A)=E(Y1−Y0∣X).\mathbb E(Y^1-Y^0\mid X,A)= \mathbb E(Y^1-Y^0\mid X). Under our allocation mechanism, the IV assumptions, and suitable conditional relevance, conditional Wald contrasts then identify the CATE following the results in the IV literature, and averaging identifies a common ATE across designs; please see our response to reviewer BrDM for details. We will add discussion clarifying this extension.

We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.

Dear Reviewer YczW,

Thank you again for your careful reading and helpful suggestions. With the rebuttal period closing soon, we would be grateful if you could let us know whether any of your original concerns remain unresolved. We would be very happy to address them further.

Given the additional experiments and discussion we have provided, we see that many of those concerns could be resolved. We hope that, if that is indeed the case, you may consider revisiting your rating. Thank you again for your time!

Thank you for the detailed clarifications and additional simulation, which helps me to understand the connection between theory and experiments better. However, I still feel comparative evidence supporting the proposed design is rather weak and limited, and the waiting-time comparison does not fully resolve the robustness issue on timing-dependent outcomes. These concerns remain substantial enough for me to maintain my current rating.

Reviewer_LT6POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This papers considers the problem: how can we learn treatment effects while also maximizing the utility of a resource-allocation method. The authors consider an approach that randomizes participants into different priority queues via their assessed risk. They then characterize: 1) Can we learn causal effects from this system, even when arrival times endogeneous, and 2) how can we trade of the efficiency of learning treatment effects with actually providing utility to to high-needs individuals. The authors develop nice theory to show that by design, many statistical guarantees that come with i.i.d data also come here, and with arrival times being endogenous, you can use instrumental variable analysis to estimate treatment effects. The authors test their theory on a housing allocation dataset.

Critical strengths
  1. The problem is well motivated. A lot of public programs ration services through priority waitlists. The idea of randomizing the queue instead of the treatment is a neat design.

  2. The theory is quite neat, especially Theorems 5 and 6. Even though treatment assignments are dependent, and the paper still shows that the DR and IV estimators attain the usual i.i.d efficiency bounds. The connection to IV is also quite nice. When arrivals are endogenous, using the queue randomization as a formula instrument is a good idea, for connecting to effect estimation.

Critical weaknesses

Overall, the biggest weakness is the relationship between the theory and the experiments.

  1. The goal of the experiments should be to show that their theory holds in practice. However, there's a few mismatches that make the experiments unsupportive of the results.
  • A critical part of Theorem 4 is that queues are run sequentially, with some having an alpha of 0, some having an alpha of 1, and one is in between. With two equal queues and a 50% budget, that gives α = [1, 0]. The experiments use [0.1, 0.9], [0.1, 0.5, 0.9], etc. at the same 50% budget. This could soften the competition for slots, which is the neat thing that the theory tackled, and the result are not supported by/supporting the theorem anymore.

  • Figure 1 is theory and not actualized experiments. The curves plot the formulas from their theory at the chosen θ and α. They show the optimizer does well according to the formula, but they didn't actually simulate what the priority queue would look like for the real data. The work should simulate the PQ to understand whether their theory holds with real experiments.

  1. The IV results for Section 5.2 are nice, but they're a bit unrealistic as known quantities are given instead of estimated values. After checking the Appendix, it seems as though both the IV estimator and the DR estimator for the exogenous setting use known nuissance functions. However, it would be interesting to see in real experiments how the bias from the DR estimator missing the hidden confounder compares to the bias from estimating nuisance function as a whole for the endogenous setting. This would help nail down that the theory is actually applicable in real applications.
Questions for authors
  1. Why do the alpha values change between the theorems and the experiments?
  2. How does performance change when using estimated nuissance functions instead of ground-truth?
  3. How does the theory line up with actually running the proposed priority queue experiment. Specifically, could you: pick a design from Figure 1, have people arrive (randomly or using real dates), draw people from the queue, perform the waitlist, record outcomes, and then analyze what the resulting models/effects look like?
3 comments

Dear Reviewer LT6P,

Thank you for your feedback! We address your concerns below:

Why do the alpha values change between the theorems and the experiments?

Our apologies for the confusion! This distinction was insufficiently clear in our presentation. Theorem 4 characterizes the limiting design-induced treatment probabilities αk(β,p)\alpha_k(\beta,p), whereas Figure 1 evaluates the efficiency-utility trades-offs under anticipated finite-sample design-induced treatment probabilities.

In finite samples, variation in queue composition and the relative timing of arrivals and treatment releases can allow some lower-priority applicants to receive treatment while some higher-priority applicants may remain untreated at the end of enrollment. Thus, the finite-sample propensity π~n(x,k)\widetilde{\pi}_{n}(x, k) need not equal its limit αk(β,p)\alpha_k(\beta, p).

Using limiting probabilities for planning may lead to overly optimistic efficiency and sample size calculations. To assess the practical implications for sample-size planning, we fixed a queue assignment policy targeting 98% of the maximum anticipated utility and calculate the enrollment required for 80% power to detect a treatment effect of −0.1-0.1. We compared forecasts based on limiting versus anticipated finite-sample probabilities. For each recommended sample size, we ran 1,000 repetitions of the full arrival and priority-queue allocation process with a 50% treatment budget and actual queue treatment probabilities matching the planning probabilities. The result can be found below:

Assumed ratesRequired nnPredicted SDEmpirical SDPowerCoverage
(1,0)(1,0)1,5260.03570.046860.0%94.4%
(0.9,0.1)(0.9,0.1)2,3780.03570.034382.6%95.4%

Using the limiting probabilities underestimates the finite-sample variance and recommends insufficient enrollment. Therefore, for planning, we recommend practitioners use the finite-sample design-induced treatment probabilities π~n\widetilde \pi_n. After the experiment, actual treatment probabilities may be estimated from observed queue treatment offers. We will add this discussion and use separate notation for the anticipated probabilities used in Figure 1.

How does performance change when using estimated nuisance functions instead of ground-truth?

How does the theory line up with actually running the proposed priority queue experiment. Specifically, could you: pick a design from Figure 1, have people arrive (randomly or using real dates), draw people from the queue, perform the waitlist, record outcomes, and then analyze what the resulting models/effects look like?

Thank you for raising these great questions! We conducted a new simulation addressing both concerns by simulating the full arrival and queue allocation process and comparing estimators using oracle versus estimated nuisance functions. The IV estimator exhibits negligible bias and coverage close to the nominal 95% level in both cases. Please see our first response to Reviewer BrDM for the simulation setup and detailed results.

We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.

Thank you to the authors for the rebuttal. The new simulations with estimated nuisance functions answer my question there. Based on the new results, I have a few follow-up questions:

  1. I appreciate the explanation of why finite-sample treatment probabilities can differ from the limits in Theorem 4. However, the gaps in Figure 1 look much larger than noise from a finite system would explain. For five equal queues with a 50% budget, Theorem 4 gives roughly [100%, 100%, 50%, 0%, 0%], while Figure 1 uses [90%, 75%, 50%, 25%, 10%]. How were these values chosen? The gap between the theory and simulated practice still seem quite far.

  2. I appreciate that the full arrival-and-queue process was simulated. However, the budget release process was constructed to produce the target α and did not use the real dates. Could you estimate α using the program’s real arrival and unit-availability dates, and repeat the simulation under that timeline? This would show which α are realistic for this program and fulfull the goal of simulating the PQ on real data

The nuisance-function question is resolved, but the gap between the theory and the experimental setup remains my main concern. Clarifications or answers to the questions above would help me think about reconsidering my score.

Thank you for your response and careful reading of our rebuttal! We address each point below:

  1. The treatment probabilities in Figure 1 were selected as illustrative configurations rather than calibrated estimates for those specific queue designs. Their selection was motivated by our public housing dataset, in which multiple queues are partially served. With K=10K=10 historical queues and an overall treatment budget of approximately 30%, we observe the following offer rates:
QueueReceived Offer
100.05
90.07
80.06
70.17
60.23
50.20
40.35
30.48
20.61
10.83

As we intend to deploy our framework in this program, these pre-deployment observations provide practical motivation for considering configurations with several partially served queues. For five equal queues, (0.9, 0.75, 0.5, 0.25, 0.1) preserves higher treatment probabilities for higher-priority queues and averages to the specified 50% budget. We will clarify this distinction and report these empirical rates in the revision.

Also, please note that our conclusions do not rely on a single configuration. Appendices D.3.2 and D.3.3 examine alternative budgets and queue treatment probabilities.

  1. Unfortunately, our dataset does not contain unit-availability dates or offer dates, so we cannot reconstruct the past budget-release process or replay the queue. We will make this data limitation explicit and clarify that our simulations evaluate the full priority-queue mechanism under a specified release process rather than reconstructing the program's historical operation.

We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.

Reviewer_BrDMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes randomizing applicants into priority queues to learn treatment effects while allocating limited resources. It studies what effects can be estimated and how to choose queue-assignment probabilities to balance precision with serving higher-need applicants.

Critical strengths

The problem is important and well-motivated. Randomizing queue assignment may be easier to implement than directly randomizing treatment. The distinction between exogenous and endogenous arrival times is also useful.

Critical weaknesses
  1. The variance results need a simulation check. The main figures plot the theoretical variance formulas rather than the actual variability of estimates from repeated queue simulations. Simulating the full allocation process and reporting estimation error and confidence-interval coverage would help show whether the formulas work at realistic sample sizes.
  2. The propensity scores are treated as known. The DR and IV estimators use finite-sample treatment probabilities that depend on the arrival distribution and allocation process. The paper does not explain how practitioners should compute or estimate them. Theorems 5 and 6 allow estimation of outcome and residual-variance nuisance functions, but do not account for estimated propensities, and the bias experiments use known values. If these probabilities are estimated by simulating the queue, or replaced with their limits, the paper should explain whether this affects inference.
  3. Treatment timing may matter in the intended applications. The model treats everyone who receives treatment before the horizon as receiving the same intervention, regardless of how long they waited. Earlier housing assistance could have different effects from later assistance. The paper acknowledges this limitation, but some evidence on sensitivity to timing would strengthen the application.
  4. The practical consequences of heterogeneous effects need more discussion. The paper recognizes that the weighted LATE changes with the queue-assignment policy, but the optimization and simulations focus on a common treatment effect. With heterogeneous treatment effects, different queue-assignment policies may estimate effects for different groups of applicants. It would help to explain how practitioners should compare designs when both precision and the population represented by the estimate change.
Questions for authors
  1. Do the confidence intervals have the expected coverage in simulations of the full queue process?
  2. How should treatment probabilities be estimated in practice?
  3. How sensitive are the conclusions to treatment timing?
  4. How do the effects being estimated change across queue-assignment policies when treatment effects are heterogeneous?
4 comments

Dear Reviewer BrDM,

Thank you for your feedback! We address your concerns below:

Do the confidence intervals have the expected coverage in simulations of the full queue process?

Thank you for this important question regarding the analysis of experiments! To answer this, building on the data generating process in Figure 2 where arrivals are endogenous (which may be more likely in practice), we simulated the full arrival and priority-queue allocation process with a treatment budget of β=0.5\beta = 0.5 and a design targeting 9898% of the maximum anticipated utility. We also simulated a budget release process that satisfies Assumption 2 asymptotically and yields finite-sample queue treatment probabilities of approximately (0.9,0.1)(0.9, 0.1). To address Reviewer LT6P's second concern simultaneously, we compared oracle nuisances with nuisances estimated using random forests on independent training samples, as specified in Theorems 5 and 6.

We used 1,000 repetitions with nn training samples and nn evaluation samples, where n=500,1000,2000n=500,1000,2000. Below, we report the bias, the empirical standard deviation of the estimated effects (SD), the average asymptotic variance Vψ(θ,η)V_\psi(\theta,\eta), and the estimated variance Var^(ψ^)\widehat{\mathrm{Var}}(\widehat \psi) using the EIF as well as the coverage of both the asymptotic and estimated variances.

n=500n = 500:

EstimatorNuisancesBiasSD\text{SD}Var(ψ^)/n\sqrt{\mathrm{Var(\widehat \psi})/n}CoverageVar^(ψ^)/n\sqrt{\widehat{\mathrm{Var}}(\widehat \psi)/n}Coverage (EIF)
ψ^Z⋆\widehat{\psi}_Z^\starOracle-0.00530.08100.07930.94000.08200.9470
ψ^ATEDR\widehat{\psi}_{\mathrm{ATE}}^{\mathrm{DR}}Oracle0.06460.05680.05380.76700.05720.8210
ψ^Z⋆\widehat{\psi}_Z^\starEstimated-0.00510.08260.08030.93700.08290.9450
ψ^ATEDR\widehat{\psi}_{\mathrm{ATE}}^{\mathrm{DR}}Estimated0.05830.05640.05640.82500.05760.8330

n=1000n = 1000:

EstimatorNuisancesBiasSD\text{SD}Var(ψ^)/n\sqrt{\mathrm{Var(\widehat \psi})/n}CoverageVar^(ψ^)/n\sqrt{\widehat{\mathrm{Var}}(\widehat \psi)/n}Coverage (EIF)
ψ^Z⋆\widehat{\psi}_Z^\starOracle-0.00130.05800.05620.95000.05710.9560
ψ^ATEDR\widehat{\psi}_{\mathrm{ATE}}^{\mathrm{DR}}Oracle0.06350.04080.03810.61800.04000.6450
ψ^Z⋆\widehat{\psi}_Z^\starEstimate-0.00130.05830.05680.95300.05770.9570
ψ^ATEDR\widehat{\psi}_{\mathrm{ATE}}^{\mathrm{DR}}Estimated0.06090.04080.03960.66000.04020.6780

n=2000n = 2000:

EstimatorNuisancesBiasSD\text{SD}Var(ψ^)/n\sqrt{\mathrm{Var(\widehat \psi})/n}CoverageVar^(ψ^)/n\sqrt{\widehat{\mathrm{Var}}(\widehat \psi)/n}Coverage (EIF)
ψ^Z⋆\widehat{\psi}_Z^\starOracle-0.00040.03940.03970.95200.03980.9470
ψ^ATEDR\widehat{\psi}_{\mathrm{ATE}}^{\mathrm{DR}}Oracle0.06110.02900.02690.40100.02800.4210
ψ^Z⋆\widehat{\psi}_Z^\starEstimated-0.00050.04020.04010.94400.04030.9430
ψ^ATEDR\widehat{\psi}_{\mathrm{ATE}}^{\mathrm{DR}}Estimated0.06020.02850.02780.43900.02810.4430

As can be seen, the IV estimator exhibits negligible bias and coverage close to the nominal level with both oracle and estimated nuisances, even at small sample sizes (in contrast, the DR estimator remains biased under endogeneous arrivals with substantial undercoverage). As per Theorem 8 in Appendix B, analogous results will hold for the DR estimator for the ATE under arrival exogeneity.

How should treatment probabilities be estimated in practice?

Thank you for raising this practical question! We first note that, under Assumption 3, the finite-sample queue-conditional treatment probabilities become π~n(x,k)=P(Z=1∣Q=k)\widetilde\pi_n(x,k)= \mathbb P(Z=1\mid Q=k) and hence are constant in xx, which implies πn(x)=∑k=1Kθk(x)π~n(x,k).\pi_n(x)=\sum_{k=1}^K\theta_k(x)\widetilde\pi_n(x,k). A simple solution therefore is to estimate the queue treatment rates directly. If we write Nk,n=∑i=1n1(Qi=k),Tk,n=∑i=1n1(Qi=k)Zi,N_{k,n}=\sum_{i=1}^n\mathbf 1(Q_i=k), \qquad T_{k,n}=\sum_{i=1}^n\mathbf 1(Q_i=k)Z_i, using the same notation from the proof of Theorem 4, we can estimate them by

\qquad \widehat{\pi}_n(x)=\sum_{k=1}^K\widehat{\widetilde{\pi}}_n(x,k)\theta_k(x).$$ which requires only allocation records and the known queue assignment probabilities. Importantly, the bounded-differences argument in Lemma 3 yields $$\sup_x\left\lvert\widehat{\pi}_n(x)-\pi_n(x)\right\rvert=O_p(n^{-1/2}).$$ We will add this discussion on estimating treatment probabilities as well as a lemma that formalizes what’s above in Appendix B. > How sensitive are the conclusions to treatment timing? This is a great question! We think that sensitivity to treatment timing is generally difficult to quantify without additional assumptions about how outcomes respond to delays. When the goal is to capture dependence of outcomes on time to treatment, the appropriate treatment variable is the treatment time $T\in\\{1,\ldots,\tau,\infty\\}$, where $T=\infty$ denotes no treatment by the horizon. After redefining the treatment appropriately, queue assignment remains a valid instrument (under a similar assumption that queue assignment effects outcomes only through the wait time to treatment). Formally characterizing and optimizing such an estimand, however, would require additional IV theory for multi-valued or even continuous treatments, which is beyond the scope of our work. Also, program evaluations in public policy settings like ones motivating our work almost always study treatment as a binary (e.g., Kling, 2007; Finkelstein, 2012). So long as practitioners are willing to make this approximation in their setting, appropriate design and analysis tools are valuable. The main approximation being made is that when treatment is modeled as continuous, different allocation policies induce different wait-time distributions. That is, the treatment effect being estimated is inevitably local to the distribution induced by a particular design. Below we also show that this limitation may be modest in practice by comparing the treatment times induced by our optimized design ($\mathrm{opt}$) with those induced by strict prioritization ($\mathrm{str}$), which is the most likely policy-relevant alternative that we may wish to extrapolate effect estimates to. We simulate risk scores from the fitted distribution \$ h(X)\sim Beta(9.25,7.54)\$ and use five equal-sized priority queues with a 50\% treatment budget. Let $Z^{\theta}$ and $W^{\theta}$ denote treatment receipt and waiting time under policy $\theta \in\{\mathrm{opt},\mathrm{str}\}$, respectively. Across 5,000 simulations, we compare the optimized endogenous-arrival policy retaining 95\%, 97.5\%, 99\% of maximum utility of strict prioritization using the conditional Kolmogorov-Smirnov distance $$\Delta \equiv \max_{w,B} \left|P(W^{\mathrm{opt}}\leq w \mid Z^{\mathrm{opt}}=Z^{\mathrm{str}}=1,h(X)\in B) - P(W^{\mathrm{str}}\leq w \mid Z^{\mathrm{opt}}=Z^{\mathrm{str}}=1,h(X)\in B)\right|,$$ which measures the largest difference between the optimized and strict-prioritization waiting-time CDFs for applicants treated under both policies. | relative utility | $\Delta$ | jointly treated | | ---: | ---: | ---: | | 95.0% | 5.33 | 77.78% | | 97.5% | 4.93 | 84.51% | | 99.0% | 4.08 | 90.26% | As can be seen, the maximum discrepancy across all risk-score deciles and waiting-time thresholds is 5.33 percentage points for the optimized design retaining 95\% of strict-prioritization utility. Thus, among applicants who would receive treatment in both the experimental and deployment regimes, the optimized design induces a similar waiting-time distribution conditional on assessed risk. We will add explicit discussion of treatment timing to the paper! - Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J., Newhouse, J. P., Allen, H., Baicker, K., and the Oregon Health Study Group. (2012). The Oregon Health Insurance Experiment: Evidence from the first year. The Quarterly Journal of Economics. - Kling, J. R., Liebman, J. B., and Katz, L. F. (2007). Experimental analysis of neighborhood effects. Econometrica.

How do the effects being estimated change across queue-assignment policies when treatment effects are heterogeneous?

Thank you for this important question! As you mention, and as Theorem 2 shows, the IV estimand generally depends on the queue assignment policy under heterogeneous effects, so some kind of restriction is necessary to obtain an estimand invariant to designs. We believe the PLIV model, despite its strong homogeneity assumption, offers a very natural starting point.

A potentially useful extension is not to collapse heterogeneity entirely, but simply restrict those that pertain to arrivals, following ``no interaction'' assumptions in the IV literature (Wang and Tchetgen Tchetgen, 2018; Cui and Tchetgen Tchetgen, 2020; Levis et al., 2024). For example, following Assumption 5(b) in Theorem 1 of Wang and Tchetgen Tchetgen (2018), also presented in Proposition 3 of Levis et al. (2024), we can consider E(Y1−Y0∣X,A)=E(Y1−Y0∣X),\mathbb E(Y^1-Y^0\mid X,A)= \mathbb E(Y^1-Y^0\mid X), which says that mean treatment gains do not vary with AA after conditioning on XX and permits heterogeneous effects and endogenous arrivals through baseline outcomes. Thus conditional Wald contrasts identify the CATE and, provided that conditional relevance holds, averaging identifies a common ATE across designs. A caveat, however, is that an influence function (IF) for this ATE obtained through the IV/average-Wald route generally differs from the usual IF, so developing corresponding design objective and inference procedure under queue dependence is not straightforward.

We will add a discussion of these weaker restrictions and their implications and emphasize this as an important direction for future research.

  • Cui, Y. and Tchetgen Tchetgen, E. (2021), A semiparametric instrumental variable approach to optimal treatment regimes under endogeneity. Journal of the American Statistical Association.

  • Levis, A., Keele, L., Kennedy, E. H. Nonparametric identification and efficient estimation of causal effects with instrumental variables.

  • Wang, L. Tchetgen-Tchetgen, E. Bounded, Efficient and Multiply Robust Estimation of Average Treatment Effects Using Instrumental Variables. Journal of the Royal Statistical Society: Series B.

We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.

Dear Reviewer BrDM,

Thank you again for your careful reading and helpful suggestions. With the rebuttal period closing soon, we would be grateful if you could let us know whether any of your original concerns remain unresolved. We would be very happy to address them further.

Given the additional results and experiments we have provided, we see that many of those concerns could be resolved. We hope that, if that is indeed the case, you may consider revisiting your rating. Thank you again for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.