- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a new randomized assignment procedure for estimating treatment effects under resource constraints. It derives causal identification results under exogenous and endogenous arrivals and optimizes queue assignments to balance statistical efficiency and allocation utility.
Critical strengths
The motivation is interesting and practically relevant. The theoretical treatment of dependent allocations and the integration of causal inference with constrained experimental design are potentially valuable.
Critical weaknesses
-
My main concern lies in insufficient design justification. The paper does not convincingly explain why randomizing priority-queue assignments is preferable to alternative designs, such as stratified lotteries or randomization within priority tiers, under comparable operational and fairness constraints.
-
Also, comparisons are primarily against two heuristic queue-assignment methods and an RCT baseline. Since the proposed method explicitly optimizes the evaluation objective, outperforming these heuristics provides limited evidence of a substantive methodological advantage.
-
The framework assumes outcomes are unaffected by treatment timing or queue placement, which may be unrealistic in public service settings where waiting times can directly influence outcomes.
-
I think theory and experiments are somewhat disconnected as well. The experiments prescribe queue-specific treatment probabilities without sufficiently demonstrating that these correspond to the dynamic priority-queue mechanism analyzed theoretically. Moreover, the empirical analysis relies heavily on semi-synthetic outcomes with homogeneous treatment effects. Under endogenous arrivals, the IV estimand can depend on the design itself, making comparisons under treatment effect heterogeneity less straightforward.
Questions for authors
See my comments in the weakness section
4 comments
I think theory and experiments are somewhat disconnected as well. The experiments prescribe queue-specific treatment probabilities without sufficiently demonstrating that these correspond to the dynamic priority-queue mechanism analyzed theoretically. Moreover, the empirical analysis relies heavily on semi-synthetic outcomes with homogeneous treatment effects. Under endogenous arrivals, the IV estimand can depend on the design itself, making comparisons under treatment effect heterogeneity less straightforward.
Thank you for raising these concerns, and our apologies for the confusion! As we explain in our response to Reviewer LT6P, Theorem 4 characterizes the limiting design-induced treatment probabilities, whereas Figure 1 evaluates efficiency–utility trade-offs under prescribed queue-specific treatment rates. In finite samples, variation in queue composition and the relative timing of arrivals and treatment releases can allow some lower-priority applicants to receive treatment while some higher-priority applicants remain untreated. Thus, the finite-sample treatment probabilities need not equal their limits.
To demonstrate the connection to the dynamic allocation mechanism, we conducted a new simulation of the full arrival and priority-queue allocation process, building on the endogenous-arrival setting in Figure 2. We used a treatment budget of 50% and a design targeting 98% of maximum anticipated utility. The budget release process satisfies Assumption 2 asymptotically and yields finite-sample queue treatment probabilities of approximately (0.9, 0.1). Across 1,000 repetitions at sample sizes of 500, 1,000, and 2,000, the IV estimator exhibits negligible bias and coverage close to the nominal 95% level with both oracle and estimated nuisances. Please refer to our first response to Reviewer BrDM for details. We will incorporate this experiment and distinguish the planning rates from the limiting probabilities more explicitly.
Regarding heterogeneous effects, as you mention and as Theorem 2 shows, the IV estimand generally depends on the queue-assignment policy under unrestricted heterogeneity. Consequently, some kind of restriction is necessary to obtain an estimand invariant to designs. We believe the homogeneous PLIV model, despite imposing a strong additive homogeneity restriction, offers a natural starting point.
A useful extension is to only restrict arrival-relevant heterogeneity through a ``no interaction'' assumption such as Under our allocation mechanism, the IV assumptions, and suitable conditional relevance, conditional Wald contrasts then identify the CATE following the results in the IV literature, and averaging identifies a common ATE across designs; please see our response to reviewer BrDM for details. We will add discussion clarifying this extension.
We thank the reviewer again for their time and consideration. We are more than happy to address any other concerns they might have.
Dear Reviewer YczW,
Thank you again for your careful reading and helpful suggestions. With the rebuttal period closing soon, we would be grateful if you could let us know whether any of your original concerns remain unresolved. We would be very happy to address them further.
Given the additional experiments and discussion we have provided, we see that many of those concerns could be resolved. We hope that, if that is indeed the case, you may consider revisiting your rating. Thank you again for your time!
Thank you for the detailed clarifications and additional simulation, which helps me to understand the connection between theory and experiments better. However, I still feel comparative evidence supporting the proposed design is rather weak and limited, and the waiting-time comparison does not fully resolve the robustness issue on timing-dependent outcomes. These concerns remain substantial enough for me to maintain my current rating.
Dear Reviewer YczW,
Thank you for your feedback! We address your concerns below:
Thank you for giving us this opportunity to elaborate more on the alternatives! As you point out, one natural alternative is to assign applicants deterministically to priority tiers using, for instance, cutoffs on estimated risk scores, and then randomize within each tier. Such designs, however, can raise ethical concerns in a waitlist setting, as we discuss in our paper. If randomization determines in advance that an applicant will not receive treatment during the allocation horizon, we believe that decision should be communicated promptly. Keeping that person on an apparently active waitlist despite their predetermined exclusion can delay their pursuit of other options and impose avoidable waiting costs.
Repeated lotteries that maintain everyone’s eligibility could address the concern above. This approach, however, replaces FIFO within each queue and therefore allows later arrivals to receive treatment ahead of applicants who have waited longer, changing an important procedural-fairness feature of systems where waiting time establishes priority.
We therefore view our design as a practical middle ground that addresses these concerns and respects existing operational systems and constraints, while introducing randomness that allows program administrators to learn treatment effects. We will augment the Introduction to explain these alternatives and their ethical and operational considerations more explicitly!
We appreciate the constructive criticism! We however would like to emphasize that, to our knowledge, no existing experimental design method jointly addresses the trade-off between prioritization and randomization, sequential arrivals, and resource availability. Wilder and Welle (2025), the work we build off of, consider a static setting without these sequential components. Establishing a principled design approach under these constraints is a central methodological contribution of our work.
In particular, our contribution is not simply applying an optimizer to an already established objective. As can be seen, we characterize how queue randomization induces treatment probabilities, justify an efficiency criterion despite the resulting dependent treatment assignments, and obtain a tractable constrained optimization problem. Setting the ethical concerns aside, the alternative allocation mechanisms mentioned above would require additional analysis to yield a principled experimental design. They are therefore not directly available as comparable existing methods, which motivates our comparison with natural heuristics that practitioners might otherwise use.
Thank you for raising this important question! We agree that waiting times can plausibly affect outcomes. As we explain in our response to Reviewer BrDM, when the goal is to capture dependence of outcomes on time to treatment, the appropriate treatment variable is the treatment time. After redefining treatment appropriately, queue assignment can remain a valid instrument, provided it affects outcomes only through this expanded treatment. Formally characterizing and optimizing such an estimand, however, would require additional assumptions and IV theory for multivalued or continuous treatments that go beyond the scope of our work. Further, program evaluations in public-policy settings like those motivating our work often model treatment as binary; we therefore believe our design and analysis tools are still valuable when this approximation is appropriate.
We nevertheless compare treatment times under our optimized design and strict prioritization as a descriptive check. Among applicants treated under both policies, the maximum waiting-time CDF discrepancy across risk-score deciles is 5.33%, which is modest. Please refer to our response to Reviewer BrDM for details; we will add explicit discussion of this limitation.