- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper finds the gap that, for the prediction-driven allocation models, existing timing results characterize when to intervene given the population’s risk distribution, but do not address whether a short observation period reveals enough about that distribution to determine whether acting now or waiting would lead to better outcomes. The paper closes this gap, showing that observations can reveal the population’s risks and whether it can choose the best intervention date. The paper also finds that uncertainty about the best time to intervene can persist even when the number of people observed becomes arbitrarily large.
Critical strengths
- The problem definition is clear and useful. The paper considers the prediction-driven allocation models, which appear to be useful in multiple real-world application settings.
- The experimental setting and results are clear. The findings are useful and interesting. The result of the experiment supports the claim that early stopping can outperform a converged likelihood fit, as claimed in the contribution. The claimed contributions are verified by the experiment.
- There appears to be no mathematical errors for the Lemmas, Propositions, and Theorems proposed in the paper.
Critical weaknesses
My concerns majorly focus on the theoretical part. Although the findings are interesting, I would expect to see a theoretical explanation of the findings. To make the paper complete and persuasive, I suggest that a complete theory should be included to explain the claim of line 62-67.
Besides, I have some additional concerns.
- The proposed theoretical results are limited and not significant. As an example to my major concern, the theoretical analysis focuses on .
- The clarity of the definitions needs improvement. For example, please define clearly in the main text when it first appears.
- The presentation structure needs improvement. I hope to see more real-world application examples besides the hospital example. I would also expect to see more interpretation of the lemmas and theorems. For example, a further interpretation (or a discussion including which and the regret upper bound) of Theorem 1 would be better.
Questions for authors
- Are there any theoretical results for general ? I would expect more theoretical results.
- The author claimed that ‘Comparisons holding information fixed show that early stopping can outperform a converged likelihood fit, while regret weighting offers no improvement over ordinary averaging in the reported pooled results.’ I wonder if there is a theoretical explanation for this?
- Did the author consider that how does the proposed lower bound related to the experiment results? Is there any theory for the methods’ regret?
- How can the findings of this paper be interpreted to the real-world application settings?
3 comments
Dear Reviewer KNZD,
Thank you again for your review. As the discussion period draws to a close, we wanted to follow up on our response to your theoretical concerns.
The revision adds Corollary 1, which shows that unavoidable regret is zero if and only if the reference optimal date is optimal for every compatible population; this is the exact characterization behind lines 62–67. It also extends Proposition 2, Corollary 1, and Lemma 1 to general γ (Appendix A.5), bounds the asymptotic worst-case regret of consistent plug-in rules (Appendix A.6), and compares each method against exact minimax regret on 30 compatible pairs (Appendix D.3).
We would be grateful to know whether these additions answer your questions, and we are happy to clarify anything further before the deadline.
Dear Reviewer KNZD,
We have provided detailed responses to your theoretical concerns and followed up, but have not received a reply. With 24 hours remaining in the discussion period, we ask you to assess the revisions and indicate whether any concerns remain.
The revision directly addresses your main requests:
- Decision-identification theory: New Corollary 1 establishes, under a unique reference optimum, that unavoidable regret is zero if and only if that date is optimal for every compatible population, providing the characterization requested for lines 62-67.
- Scope of γ: The submitted paper already explicitly restricted its main analysis to γ = 1; this was a stated modeling assumption, not an overlooked case. This setting is central to the motivating problem. Warnings and failures depend on the same underlying individual risk, allowing us to isolate the tradeoff between learning about risk and losing intervention opportunities. General γ introduces a different relationship between warning signals and failure risk; greater generality does not by itself make that setting more relevant to the question studied. Nevertheless, Appendix A.5 now extends Proposition 2 and Corollary 1 to general γ and Lemma 1 through a modified observable span.
- Empirical explanation and interpretation: Section 7.2 and Appendix D.4 examine initialization sensitivity and held-out-likelihood stopping. We distinguish the empirical early-stopping advantage from proven guarantees. Sections 1 and 4.3 also expand the applications and operational interpretation. Your current assessment predates these additions. Please review the revised analysis and provide an updated assessment before discussion closes, or identify which specific theoretical concern remains unresolved so that we can respond.
Thank you for your time.
We thank you for your careful review and for articulating what the theory should explain. We have added a decision-identification characterization, extensions beyond γ=1, and theoretical and empirical regret analyses.
Main concern: Lines 62–67
Observations can leave the population ambiguous while determining an optimal intervention date. New Corollary 1 makes this precise: assuming a unique reference optimum, unavoidable regret is zero if and only if that date is optimal for every compatible population. Otherwise, observationally indistinguishable populations require different decisions, forcing positive worst-case regret at every sample size.
This gives an exact characterization, though not a closed-form parameter condition. Section 5.1 certificates establish a common optimum in 1, 16, and 27 of 30 configurations at τ=2,3,4, respectively; Appendix B.3 bounds regret in ambiguous cases.
W1 and Q1: General γ
Revised Appendix A.5 extends Proposition 2 and Corollary 1 to every γ, and Lemma 1 through a modified observable span.
New Proposition A.1 establishes, for infinite-support reference populations, that hτ is identified for reduced rational γ=a/b iff a+b≤τ+1; irrational γ prevents identification. The abstract decision characterization therefore generalizes, while explicit general-γ welfare/date identification conditions remain open. Rational γ permits polynomial certificates in principle, but this extension is unimplemented; irrational-γ certificates remain open.
W2: Definition of Wt(P;β)
Section 3 now defines this quantity in words and by formula at first use: expected benefit per initial individual when spending the budget at date t, prioritizing cohorts with more warnings and replacing cohort sizes by their expectations. Appendix E adds a notation table.
W3 and Q4: Interpretation and applications
Section 1 adds homelessness and dropout prevention as illustrative, unvalidated mappings: warnings are service-record or attendance flags; failure is homelessness onset or leaving school.
Section 4.3 explains Proposition 2: indistinguishable populations requiring different dates force regret under at least one population. Compatible pairs provide lower bounds; certificates upper-bound a chosen date’s worst-case loss. Longer observation can clarify decisions, but waiting may miss the best date (Sec. 8).
Q3: Experimental comparisons and method guarantees
Appendix D.3 compares methods against exact minimax regret on 30 compatible pairs, a benchmark at least as large as the two-point lower bound. Early-stopped EM is closest at N=103; nine of eleven methods have reported ratios of 1.24–1.52 at N=105. With 2–5 trials per pair, this does not establish finite-sample near-optimality.
Appendix A.6 bounds asymptotic worst-case regret for consistent plug-in rules, including certified NPMLE, by the largest Lτ(d) among dates optimal for some compatible population. Fixed-iteration EM, bagging, Beta, and finite-sample guarantees remain uncovered.
Q2: Early stopping and regret weighting
Likelihood convergence need not improve decisions: equally good fits select different dates in 5 of 30 configurations without sampling noise.
Early stopping’s advantage remains empirical. Third-moment bias may discourage premature intervention, but targeted tests did not confirm this mechanism; the bias can also hurt. Held-out-likelihood stopping outperforms certified NPMLE (Sec. 7.2; App. D.4).
Regret weighting shows no improvement over ordinary averaging in pooled results, and we establish no theoretical advantage.
We sincerely thank you for the constructive feedback, which has been very instrumental in strengthening the foundations and theory of our work. If our additions answer your theoretical concerns, we would kindly ask you to consider raising your score.