acceptodds
Under review as a conference paper at ICLR 2027

Waiting Buys Exactly Two Moments: Endogenous Identification in Prediction-Driven Allocation

Abstract

A planner with a fixed budget of preventive help must decide whom to help and when. Waiting reveals risk but removes intervention opportunities as higher-risk individuals leave first. We characterize this tradeoff under repeated noisy signals and risk-dependent attrition: in the canonical model, a length- observation prefix identifies exactly the latent risk moments for , buying two new moments per additional period. These identify average next-period attrition among survivors but need not identify allocation welfare or attrition among those prioritized for treatment. Our observationally equivalent constructions establish unavoidable timing regret at any sample size for a fixed observation length, while exact certificates determine whether all compatible populations share a unique optimal date. Matched experiments separate the effects of estimation, bootstrap averaging, and regret weighting. Longer observation often resolves timing ambiguity, but the optimal date may already have passed. In our experiments, bootstrap averaging and early stopping improve decisions, while regret weighting adds no improvement in pooled mean squared regret: better statistical fit need not yield better allocation welfare.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
prediction-driven allocation, allocation timing, partial identification, selective attrition, moment identification, statistical decision theory

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_KNZDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper finds the gap that, for the prediction-driven allocation models, existing timing results characterize when to intervene given the population’s risk distribution, but do not address whether a short observation period reveals enough about that distribution to determine whether acting now or waiting would lead to better outcomes. The paper closes this gap, showing that observations can reveal the population’s risks and whether it can choose the best intervention date. The paper also finds that uncertainty about the best time to intervene can persist even when the number of people observed becomes arbitrarily large.

Critical strengths
  1. The problem definition is clear and useful. The paper considers the prediction-driven allocation models, which appear to be useful in multiple real-world application settings.
  2. The experimental setting and results are clear. The findings are useful and interesting. The result of the experiment supports the claim that early stopping can outperform a converged likelihood fit, as claimed in the contribution. The claimed contributions are verified by the experiment.
  3. There appears to be no mathematical errors for the Lemmas, Propositions, and Theorems proposed in the paper.
Critical weaknesses

My concerns majorly focus on the theoretical part. Although the findings are interesting, I would expect to see a theoretical explanation of the findings. To make the paper complete and persuasive, I suggest that a complete theory should be included to explain the claim of line 62-67.

Besides, I have some additional concerns.

  1. The proposed theoretical results are limited and not significant. As an example to my major concern, the theoretical analysis focuses on γ=1\gamma=1.
  2. The clarity of the definitions needs improvement. For example, please define Wt(P;β)W^t(\mathcal{P;\beta}) clearly in the main text when it first appears.
  3. The presentation structure needs improvement. I hope to see more real-world application examples besides the hospital example. I would also expect to see more interpretation of the lemmas and theorems. For example, a further interpretation (or a discussion including which and the regret upper bound) of Theorem 1 would be better.
Questions for authors
  1. Are there any theoretical results for general γ\gamma? I would expect more theoretical results.
  2. The author claimed that ‘Comparisons holding information fixed show that early stopping can outperform a converged likelihood fit, while regret weighting offers no improvement over ordinary averaging in the reported pooled results.’ I wonder if there is a theoretical explanation for this?
  3. Did the author consider that how does the proposed lower bound related to the experiment results? Is there any theory for the methods’ regret?
  4. How can the findings of this paper be interpreted to the real-world application settings?
3 comments

We thank you for your careful review and for articulating what the theory should explain. We have added a decision-identification characterization, extensions beyond γ=1\gamma=1, and theoretical and empirical regret analyses.

Main concern: Lines 62–67

Observations can leave the population ambiguous while determining an optimal intervention date. New Corollary 1 makes this precise: assuming a unique reference optimum, unavoidable regret is zero if and only if that date is optimal for every compatible population. Otherwise, observationally indistinguishable populations require different decisions, forcing positive worst-case regret at every sample size.

This gives an exact characterization, though not a closed-form parameter condition. Section 5.1 certificates establish a common optimum in 1, 16, and 27 of 30 configurations at τ=2,3,4\tau=2,3,4, respectively; Appendix B.3 bounds regret in ambiguous cases.

W1 and Q1: General γ\gamma

Revised Appendix A.5 extends Proposition 2 and Corollary 1 to every γ\gamma, and Lemma 1 through a modified observable span.

New Proposition A.1 establishes, for infinite-support reference populations, that hτh_\tau is identified for reduced rational γ=a/b\gamma=a/b iff a+b≤τ+1a+b\leq\tau+1; irrational γ\gamma prevents identification. The abstract decision characterization therefore generalizes, while explicit general-γ\gamma welfare/date identification conditions remain open. Rational γ\gamma permits polynomial certificates in principle, but this extension is unimplemented; irrational-γ\gamma certificates remain open.

W2: Definition of Wt(P;β)W^t(\mathcal P;\beta)

Section 3 now defines this quantity in words and by formula at first use: expected benefit per initial individual when spending the budget at date tt, prioritizing cohorts with more warnings and replacing cohort sizes by their expectations. Appendix E adds a notation table.

W3 and Q4: Interpretation and applications

Section 1 adds homelessness and dropout prevention as illustrative, unvalidated mappings: warnings are service-record or attendance flags; failure is homelessness onset or leaving school.

Section 4.3 explains Proposition 2: indistinguishable populations requiring different dates force regret under at least one population. Compatible pairs provide lower bounds; certificates upper-bound a chosen date’s worst-case loss. Longer observation can clarify decisions, but waiting may miss the best date (Sec. 8).

Q3: Experimental comparisons and method guarantees

Appendix D.3 compares methods against exact minimax regret on 30 compatible pairs, a benchmark at least as large as the two-point lower bound. Early-stopped EM is closest at N=103N=10^3; nine of eleven methods have reported ratios of 1.24–1.52 at N=105N=10^5. With 2–5 trials per pair, this does not establish finite-sample near-optimality.

Appendix A.6 bounds asymptotic worst-case regret for consistent plug-in rules, including certified NPMLE, by the largest Lτ(d)L_\tau(d) among dates optimal for some compatible population. Fixed-iteration EM, bagging, Beta, and finite-sample guarantees remain uncovered.

Q2: Early stopping and regret weighting

Likelihood convergence need not improve decisions: equally good fits select different dates in 5 of 30 configurations without sampling noise.

Early stopping’s advantage remains empirical. Third-moment bias may discourage premature intervention, but targeted tests did not confirm this mechanism; the bias can also hurt. Held-out-likelihood stopping outperforms certified NPMLE (Sec. 7.2; App. D.4).

Regret weighting shows no improvement over ordinary averaging in pooled results, and we establish no theoretical advantage.

We sincerely thank you for the constructive feedback, which has been very instrumental in strengthening the foundations and theory of our work. If our additions answer your theoretical concerns, we would kindly ask you to consider raising your score.

Dear Reviewer KNZD,

Thank you again for your review. As the discussion period draws to a close, we wanted to follow up on our response to your theoretical concerns.

The revision adds Corollary 1, which shows that unavoidable regret is zero if and only if the reference optimal date is optimal for every compatible population; this is the exact characterization behind lines 62–67. It also extends Proposition 2, Corollary 1, and Lemma 1 to general γ (Appendix A.5), bounds the asymptotic worst-case regret of consistent plug-in rules (Appendix A.6), and compares each method against exact minimax regret on 30 compatible pairs (Appendix D.3).

We would be grateful to know whether these additions answer your questions, and we are happy to clarify anything further before the deadline.

Dear Reviewer KNZD,

We have provided detailed responses to your theoretical concerns and followed up, but have not received a reply. With 24 hours remaining in the discussion period, we ask you to assess the revisions and indicate whether any concerns remain.

The revision directly addresses your main requests:

  • Decision-identification theory: New Corollary 1 establishes, under a unique reference optimum, that unavoidable regret is zero if and only if that date is optimal for every compatible population, providing the characterization requested for lines 62-67.
  • Scope of γ: The submitted paper already explicitly restricted its main analysis to γ = 1; this was a stated modeling assumption, not an overlooked case. This setting is central to the motivating problem. Warnings and failures depend on the same underlying individual risk, allowing us to isolate the tradeoff between learning about risk and losing intervention opportunities. General γ introduces a different relationship between warning signals and failure risk; greater generality does not by itself make that setting more relevant to the question studied. Nevertheless, Appendix A.5 now extends Proposition 2 and Corollary 1 to general γ and Lemma 1 through a modified observable span.
  • Empirical explanation and interpretation: Section 7.2 and Appendix D.4 examine initialization sensitivity and held-out-likelihood stopping. We distinguish the empirical early-stopping advantage from proven guarantees. Sections 1 and 4.3 also expand the applications and operational interpretation. Your current assessment predates these additions. Please review the revised analysis and provide an updated assessment before discussion closes, or identify which specific theoretical concern remains unresolved so that we can respond.

Thank you for your time.

Reviewer_rU6COpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the problem of choosing when to allocate a limited intervention budget in a population with unknown, heterogeneous failure risks. Waiting provides additional warning signals that help identify higher-risk individuals, but also causes individuals to fail and lose eligibility for intervention. The authors show that observing τ\tau periods identifies the first 2τ−12\tau-1 moments of the risk distribution at the population level, but these moments may not suffice to identify the optimal intervention date. They establish an information-theoretic regret lower bound and develop a computational procedure to certify whether an optimal date is shared by all observationally compatible distributions. Finally, they empirically compare several estimation and decision-making methods, finding that early-stopped EM performs particularly well.

Critical strengths

Interesting problem: The tradeoff between acquiring better information and losing opportunities to intervene is compelling and relevant to real-world resource allocation.

Interesting identification perspective: Studying what can be inferred about optimal intervention timing from limited observations is a natural extension of prior work. The moment characterization is elegant.

Critical weaknesses
  1. Serious presentation issues. I found the paper extremely difficult to follow, largely because important concepts and mathematical objects are introduced without sufficient definition or intuition. For example, "mean-field welfare," which is central to the paper, is not explicitly defined in the main text and is not adequately explained even in the appendix. Similarly, "observation kernels" and Eτ[p]E_\tau[p] are introduced without clearly explaining their meaning.

More fundamentally, the abstract and introduction could make the problem formulation much clearer. The underlying problem is relatively simple: the planner does not know the population risk distribution, observes noisy signals and failures over time, and must choose a single date at which to allocate a limited intervention budget according to a fixed ranking rule. However, these essential features are not clearly communicated. The paper also introduces unnecessary notation, such as γ\gamma, despite fixing it to 1 throughout the main analysis. These issues are pervasive and made it unnecessarily difficult to understand the problem and contributions.

  1. Limited quantitative insights from the theoretical results. Theorem 1 is essentially a standard indistinguishability argument: if two observationally equivalent distributions have different optimal decisions, any algorithm must incur regret under at least one of them. While correct, the result seems relatively straightforward. I would have liked a more quantitative characterization of unavoidable regret as a function of the observation length, rather than a lower bound expressed in terms of the welfare differences of an arbitrary compatible pair.

  2. Lack of a unified contribution. I found the connection between the theoretical, computational, and empirical results unclear. The exact certification procedure bounds welfare differences but is used primarily to determine whether the optimal date is uniquely identified, rather than to provide welfare-loss guarantees for decisions made under an unknown risk distribution. Such guarantees seem more relevant to the motivating allocation problem. Meanwhile, the strongest empirical results come from early-stopped EM, an existing technique, without a theoretical explanation for its effectiveness. Overall, the different components do not form a sufficiently coherent picture of how to make better intervention decisions.

Overall Assessment

I find the problem interesting and believe the paper contains some technically worthwhile ideas. However, the substantial presentation issues, relatively limited quantitative theoretical insights, and lack of a clear connection between the theory and empirical methods substantially weaken the contribution.

Questions for authors

Please see weaknesses

5 comments

We thank you for the careful review, and we appreciate the recognition of the importance of the problem and the value of the identification perspective. Your review was very helpful!

W1. Presentation

We revised the abstract and introduction around your formulation. A planner observes noisy warnings and failures in a population with an unknown risk distribution and must choose a single date to allocate a limited budget according to a fixed ranking rule. Before introducing the results, we explain that waiting improves information but also allows individuals to fail and lose eligibility.

We also clarify the specific concepts you highlighted.

  • Mean-field welfare (Sec. 3, revised). We now define mean-field welfare in words and by formula as the expected total benefit of intervention at a given date, divided by the initial population size. We explain that the “mean-field” calculation uses the expected number of eligible individuals in each warning group rather than the actual counts, which vary randomly. This lets us isolate how uncertainty about the population’s risk distribution affects the choice of intervention date. It approximates the allocation’s benefit when the population is large.
  • Observation kernels (Sec. 4.1, revised). We explain that ℓs,k(p)\ell_{s,k}(p) is the probability that an individual with risk pp remains eligible at date ss with kk warnings. Averaging this probability over the risk distribution gives the corresponding population cohort share.
  • Survivor risk (Sec. 4.2, revised). We rename Eτ[p]E_\tau[p] to hτh_\tau and define it as the average risk among individuals still eligible at date τ\tau, equivalently their probability of failure in the next untreated period.
  • Gamma (Sec. 3, revised). We remove γ\gamma from the main text and use Bernoulli⁡(p)\operatorname{Bernoulli}(p) signals throughout, retaining the general signal model in Appendix A.5.
  • Global pass. Every symbol is defined at first use. We also add a notation table (Appendix E, revised) and a roadmap paragraph connecting the identification results, certificates, and experiments.

W2. Quantitative characterization of unavoidable regret

Thank you for asking for a quantitative characterization of how unavoidable regret changes with observation length. We agree that the original lower bound alone did not adequately answer this question. We have added lower and upper bounds for all 30 configurations at τ=2,3,4\tau=2,3,4 (Sec. 4.3 and Appendix B.3, revised). We also relabel Theorem 1 as Proposition 2 and explicitly describe its proof as a standard Le Cam two-point argument.

Section 4.3 (revised) formally defines unavoidable regret Rτ⋆R^\star_\tau and the worst-case loss Lτ(d)L_\tau(d) of committing to date dd. For every observationally compatible pair (Q0,Q1)(Q_0,Q_1), every date dd, and every sample size NN, we obtain

\leq R^\star_\tau \leq L_\tau(d).$$ The lower inequality follows from Proposition 2, expressed in absolute welfare loss. The upper inequality holds because always choosing date $d$ is one admissible decision rule. Appendix B.3 (revised) reports the bounds for every configuration. For the reported upper bounds, $d$ is the reference population’s optimal date. We bound its loss against every competing date using the branch-and-bound procedure in Sec. 5.1 (revised) and independently check each certificate with an exact verifier. | Across 30 configurations | $\tau=2$ | $\tau=3$ | $\tau=4$ | |---|---|---|---| | Configurations with an ambiguous optimal date | 29 | 14 | 3 | | Largest pair-based lower bound | 0.068 | 0.013 | 0.0048 | | Largest certified upper bound | 0.174 | 0.034 | 0.011 | | Median certified upper bound among ambiguous configurations | 0.051 | 0.0033 | 0.00029 | All values are normalized by the corresponding reference population’s optimal welfare, which differs from the relative-regret normalization in Table 1. Upper bounds are certified for all 46 ambiguous cases across $\tau=2,3,4$. At $\tau=4$, 27 of 30 configurations have zero unavoidable regret; the remaining three have certified worst-case loss at most 1.1% of reference-optimal welfare. Regret cannot increase with additional observations for a fixed candidate-date set, although the reported medians cover different ambiguous subsets. The bounds need not be tight: optimizing the population pairs or chosen date could improve them. A zero pair-based lower bound does not imply zero regret; at $\tau=3$, three ambiguous cases have zero lower bounds but certified upper bounds at most 0.34% (Appendix B.3, revised). Our contribution is therefore the exact characterization of observational equivalence in Lemma 1, together with verified constructions and certificates over the full compatible population set.

W3. Connecting theory, certificates and experiments

Thank you for highlighting this disconnect. We agree that the original presentation did not sufficiently explain how the theoretical, computational, and empirical results inform the same intervention decision. We have revised the introduction and discussion to make this connection explicit and now present the certificates as welfare-loss guarantees.

Certificates as welfare-loss guarantees (Sec. 5.1, revised). A certified optimal date guarantees zero regret relative to the best date in the comparison set, for every population consistent with the observations. Table 2 establishes these guarantees in 1, 16, and 27 configurations when comparing all dates, and in 3, 18, and 28 configurations when comparing only dates still available at τ=2,3,4\tau=2,3,4, respectively.

When optimality remains unresolved, certificates bound a choice’s worst-case loss (Appendix B.3). At τ=2\tau=2, one configuration gives loss ≥\geq 0.151 for date 1 but ≤0.141 for date 2, proving that date 2 improves the worst-case guarantee, in units of reference-optimal welfare.

Why early-stopped EM helps (Sec. 7.2, App. D.4, revised). We agree that its empirical performance requires closer examination. We added a separate study with a protocol fixed before inspecting its results. It connects the estimation results to the identification problem and tests whether early stopping remains useful without access to the true optimal date.

  • Equally good fits can imply different decisions. Certified EM endpoints obtained from different initializations fit identical observation probabilities yet select different dates in 5 of 30 configurations, even without sampling noise. This directly connects the experiments to the theory.
  • We identify a possible explanation, with explicit limits. Early-stopped EM tends to overestimate m3m_3, potentially avoiding premature intervention and higher regret. This post hoc explanation was not confirmed by prespecified tests, and the bias can hurt when earlier intervention is optimal. We therefore present it as an empirical explanation to investigate, rather than an established mechanism.
  • The benefit persists with a data-based stopping rule. Selecting the stopping point by two-fold held-out likelihood, without using the true optimal date, improves on the certified nonparametric maximum-likelihood estimator (NPMLE). The differences in MSR×103\mathrm{MSR}\times 10^3, with negative values favoring early stopping, are −3.17-3.17 (interval: [−5.31,−1.53][-5.31, -1.53]) at N=103N=10^3 and −1.34-1.34 (interval: [−2.77,−0.28][-2.77,-0.28]) at N=105N=10^5, over 50 datasets per configuration.

A unified decision framework (Secs. 1 and 8, revised). The theory establishes what waiting reveals and why the optimal date can remain unknown even with arbitrarily many records. The certificates determine whether that uncertainty matters for a particular decision and bound its welfare cost when it does. The experiments then examine how to choose a date when ambiguity remains. On our reference populations, averaging and early stopping improve decisions, while regret weighting provides no additional improvement over averaging. These empirical findings complement the identification results; they do not establish a universally optimal method.

We appreciate your feedback, which prompted clearer definitions, quantitative regret bounds, and a more direct connection between the results and the planner’s decision. We hope these revisions address your concerns and would be grateful if you would reconsider your assessment in light of the revised analysis.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary of Rebuttal

Dear Area Chair,

The discussion period closes today. Reviewers rU6C and KNZD have not posted a single comment. We posted full responses on November 13-14, followed up with both reviewers on November 16 and again on November 18, and summarized the revisions for you on November 14 and November 18. Both reviews were last modified on November 6, before any rebuttal; the scores of 1 and 2 therefore assess a manuscript that has since been substantially revised.

ICLR’s reviewer guidelines require reviewers to “state your reasoning and what did/didn’t change your recommendation throughout the discussion phase.” Neither reviewer has done so, and neither has named a concern that remains open. We cannot answer objections that are never raised.

We therefore ask that you evaluate the final revision against the six priorities you set on November 10. Each is addressed in the manuscript itself:

  • Welfare objective, observation window and decision setting (Sec. 3). Mean-field welfare and the observation kernels are defined by formula at first use. The observation window is stated explicitly: the prefix ends after the period-τ signals but before that period’s failures. The future-population and current-population settings are defined separately, with separate certificates in Table 2.
  • Contribution beyond the indistinguishability argument (Sec. 4.3). Proposition 2 is credited to Le Cam’s two-point argument. The contribution is Lemma 1’s exact characterization of observational equivalence and Corollary 1: unavoidable regret is zero if and only if the reference optimal date is optimal for every compatible population, and otherwise stays positive at every sample size.
  • Scope of the canonical model (Sec. 4.1; App. A.5). Under the general signal model, Proposition 2, inequality (1) and Corollary 1 hold unchanged. Survivor risk is identified only for rational γ = a/b with a + b ≤ τ + 1, and the certificates do not apply directly; the paper states this limit.
  • Certificates as date-selection and welfare-loss guarantees (Secs. 5.1, 7.1, 8; App. B.3). The certificates resolve every benchmark case. The optimal date is certified in 1, 16 and 27 of 30 configurations at τ = 2, 3 and 4; at τ = 4, the remaining three lose at most 1.1% of optimal welfare. App. B.3 gives a configuration where date 2 has a provably smaller worst-case loss than the reference-optimal date 1 (at most 0.141, against at least 0.151). Sec. 8 turns this into a three-step procedure for the planner.
  • Early stopping, initialization and population choice (Sec. 7.2; App. D.4). In a study whose protocol was fixed before its results were inspected, EM endpoints that fit identical observation probabilities select different dates in 5 of 30 configurations. Across eight initializations, EM at 55 iterations has MSR (×10³) of 0.14–0.33 at N = 10³, against 4.47 for certified NPMLE. It beats NPMLE in 28 of 30 configurations at N = 10³ and in 17-18 at N = 10⁵, where most of the rest are ties; every loss occurs at the largest budget, β = 0.01. Choosing the stopping point by held-out likelihood, without access to the true optimal date, still improves on NPMLE. Prespecified tests did not confirm the proposed mechanism, and the paper says so.
  • Method effects versus information, and exact versus approximate results (Secs. 5.1, 5.2, 6.2). All procedures receive the same pre-failure prefix, and the likelihood-based and moment-based families determine the same population moments (Lemma 1); cross-family comparisons therefore differ in procedure, not information. The certificates use exact population moments and do not cover sampling error. The grid-based ambiguity rules approximate the full-set quantities but do not certify them.

We kindly ask you to make our final decision based on the changes we discussed here, considering that the reviewers did not make any contributions to the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.