- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper considers a sequential hypothesis testing problem in which an agent pays the cost of observations, receives a reward upon rejection of the null hypothesis, and may quit if the cost exceeds the reward, and a principal seeks to maximize the probability of success, namely, the probability of rejecting the null hypothesis under the alternative hypothesis before the agent quits. The paper establishes that the classical likelihood ratio process may not be optimal in this setting by showing that some asymptotic results (e.g., Gaussian) where the success probability can be increased by using a tempered process.
Critical strengths
- The paper makes an interesting mathematical observation, namely, that maximizing the expected evidence growth need not maximize the probability of reaching a rejection threshold.
- The derived characterization of the abandonment exponent shows that the classical likelihood ratio process for testing can be suboptimal.
- The paper gives this insight that trading some growth in accumulated evidence for reduced fluctuations encourages the agent to continue participating and increases the principal's probability of success.
Critical weaknesses
-
My main concern is about the significance of the work, and its motivation and impact. I do not understand why this problem is important. It is not clear to me whether the formulation has any practical relevance. Do the objective of the principal or the assumptions about the agent (that it pays all the sampling cost and receives a fixed reward upon rejection) reflect a meaningful and relevant decision problem? Why is this an important mathematical formulation? The paper needs a more convincing application than the firm seeking regulatory approval for a product.
-
Another concern I have is novelty. The contribution appears to be a standard optimal stopping result rather than a fundamentally new sequential testing principle. The Bellman characterization itself is quite standard and bears resemblance to classic problems in optimal stopping. Is the main distinction merely the fact that the abandonment boundary is endogenous, i.e., determined through the agent's Bellman equation?
-
The improvement in success probability appears small in the numerical experiments. I wonder if such gains justify the use of tempered testing rules as opposed to standard likelihood ratio processes?
-
The paper makes the assumption that the true alternative hypothesis Q is known to the agent. This considerably simplifies the problem. Many problems in sequential testing do not make such an assumption (hence the need to collect observations) and the agent needs to learn the underlying distribution either implicitly or explicitly. This assumption is somewhat restrictive and may not hold in practice.
-
The paper does not give insights into the significance of the theoretical results. Exponents are stated without comments to explain to the reader.
-
There is an editorial comment after Corollary 4.3 that was overlooked by the authors "Check that the initial Bellman continuation value is strictly positive for the specified reward and sampling cost". This must be removed and/or addressed.
-
According to Remark 4.1, it appears that tempering for the considered Gaussian example is equivalent to having a time-dependent threshold on the standard likelihood ratio. This relates to my earlier comment about its close connection to optimal stopping results.
-
The testing rules used are restricted to the tempered processes, so the work does not shed much light on the optimal e-processes or optimal testing strategies. Also, the theoretical analysis is restricted to asymptotic settings. This further narrows the scope of the work.
-
In general, as I was reading the paper, I felt the results need more explanation and more insights following the statement of the theoretical findings, particularly the interplay between the evidence growth rate, fluctuations, and endogenous quitting boundary.
Questions for authors
Please refer to my questions above in "Critical Weaknesses".