acceptodds
Under review as a conference paper at ICLR 2027

Incentivized Sequential Testing

Abstract

We study sequential hypothesis testing when data collection is delegated to a risk-neutral agent who pays for each observation, receives a fixed reward upon rejection, and may quit at any time. The principal chooses an e-process to maximize the probability of rejection before the agent quits. We show that classical growth-optimal e-processes need not be optimal for this objective: tempering the process trades a small loss in evidence growth for reduced fluctuations, which lowers the risk of abandonment and can increase the probability of success. We characterize the abandonment exponent and its unique maximizer within the Gaussian tempered family, and extend the strict improvement to composite nulls under a local stability condition.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
e-process, incentivized testing, principal–agent hypothesis testing, sequential testing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_evmrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers a sequential hypothesis testing problem in which an agent pays the cost of observations, receives a reward upon rejection of the null hypothesis, and may quit if the cost exceeds the reward, and a principal seeks to maximize the probability of success, namely, the probability of rejecting the null hypothesis under the alternative hypothesis before the agent quits. The paper establishes that the classical likelihood ratio process may not be optimal in this setting by showing that some asymptotic results (e.g., Gaussian) where the success probability can be increased by using a tempered process.

Critical strengths
  • The paper makes an interesting mathematical observation, namely, that maximizing the expected evidence growth need not maximize the probability of reaching a rejection threshold.
  • The derived characterization of the abandonment exponent shows that the classical likelihood ratio process for testing can be suboptimal.
  • The paper gives this insight that trading some growth in accumulated evidence for reduced fluctuations encourages the agent to continue participating and increases the principal's probability of success.
Critical weaknesses
  • My main concern is about the significance of the work, and its motivation and impact. I do not understand why this problem is important. It is not clear to me whether the formulation has any practical relevance. Do the objective of the principal or the assumptions about the agent (that it pays all the sampling cost and receives a fixed reward upon rejection) reflect a meaningful and relevant decision problem? Why is this an important mathematical formulation? The paper needs a more convincing application than the firm seeking regulatory approval for a product.

  • Another concern I have is novelty. The contribution appears to be a standard optimal stopping result rather than a fundamentally new sequential testing principle. The Bellman characterization itself is quite standard and bears resemblance to classic problems in optimal stopping. Is the main distinction merely the fact that the abandonment boundary is endogenous, i.e., determined through the agent's Bellman equation?

  • The improvement in success probability appears small in the numerical experiments. I wonder if such gains justify the use of tempered testing rules as opposed to standard likelihood ratio processes?

  • The paper makes the assumption that the true alternative hypothesis Q is known to the agent. This considerably simplifies the problem. Many problems in sequential testing do not make such an assumption (hence the need to collect observations) and the agent needs to learn the underlying distribution either implicitly or explicitly. This assumption is somewhat restrictive and may not hold in practice.

  • The paper does not give insights into the significance of the theoretical results. Exponents are stated without comments to explain to the reader.

  • There is an editorial comment after Corollary 4.3 that was overlooked by the authors "Check that the initial Bellman continuation value is strictly positive for the specified reward and sampling cost". This must be removed and/or addressed.

  • According to Remark 4.1, it appears that tempering for the considered Gaussian example is equivalent to having a time-dependent threshold on the standard likelihood ratio. This relates to my earlier comment about its close connection to optimal stopping results.

  • The testing rules used are restricted to the tempered processes, so the work does not shed much light on the optimal e-processes or optimal testing strategies. Also, the theoretical analysis is restricted to asymptotic settings. This further narrows the scope of the work.

  • In general, as I was reading the paper, I felt the results need more explanation and more insights following the statement of the theoretical findings, particularly the interplay between the evidence growth rate, fluctuations, and endogenous quitting boundary.

Questions for authors

Please refer to my questions above in "Critical Weaknesses".

Reviewer_AF9FOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies sequential hypothesis testing in a principal–agent setting where a risk-neutral agent pays a per-sample cost, receives a fixed reward upon rejection, and may quit at any time. It shows that classical growth-optimal e-processes need not maximize the principal’s probability of rejection before the agent quits. In the Gaussian model, the paper establishes an improvement from slight tempering and characterizes the abandonment exponent under the stated asymptotic regime, with an identified unique maximizer. This improvement extends to composite nulls under a local stability condition.

Critical strengths

The paper shows that slight tempering of growth-rate optimal e-processes can improve the probability of success before the agent quits in the stated asymptotic regime. It characterizes the Gaussian abandonment exponent and extends the improvement to composite nulls under a local stability condition.

Critical weaknesses
  1. The novelty and practical motivation of tempering e-processes when the agent may quit need clearer justification. Grünwald, de Heide, and Koolen (2024) already discuss drawdowns of growth-optimal e-values and suggest considering low-variance alternatives, while McClellan (2022) studies approval design when a cost-bearing agent may quit. The novelty therefore appears to center on the abandonment-exponent analysis under the proposed model. The assumptions that the firm knows the true alternative and follows an optimal quitting rule also require stronger justification. It remains unclear how reasonable these assumptions are in the motivating application and whether the improvement persists under deviations from them.

  2. The presentation is poor. The manuscript appears to deviate from the standard ICLR template, and the lack of typographic emphasis, such as bold theorem labels, makes formal statements harder to read. The theorem statements contain substantial redundancy and lack sufficient heuristic explanations of their main conclusions and underlying mechanisms. Additional deviations from the ICLR formatting requirements are noted in the ethics section below.

Questions for authors
  1. The principal’s objective of maximizing rejection probability before the agent quits also needs stronger motivation from concrete applications. Could the authors provide more justifications?

  2. Can the local stability condition be relaxed? Additional heuristic explanation of this condition and its role in the improvement guarantee would be helpful.

  3. For a general specified Q, what is the computational cost of constructing the oracle e-process and solving the optimal quitting problem? Are the gains robust to approximating normalizing constants and transition expectations?

  4. The submission does not appear to include code. Could the authors provide anonymized code and instructions for reproducing the experiments during the rebuttal period?

Reviewer_uvS3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors consider a situation where the main agent (or the principal) is not collecting data itself, but is delegating the data collection to another agent. The data-collecting agent incurs a cost for each observation but receives a reward if the null hypothesis is rejected. However, the data-collecting agent can choose to quit at any time. So, the motivations of the two agents are different. The principal wants to gather enough evidence to reject the null hypothesis while the data-collecting agent simply wants to maximize the cumulative reward. In this setup, the authors first show in the case of a simple Gaussian null and a Gaussian alternate, using likelihood ratio can be sub-optimal, under certain conditions and tempered likelihood-ratio works better. Then they extend it to the case when the null is composite but convex. This is also supported by simulated numerical experiments.

Critical strengths

The paper discusses a probable extension of sequential testing problem. The paper shows suboptimality of classic LRT in this setting and proposes a tempered LRT solution. The solution is instantiated for Gaussian and Bernoulli distributions. Numerical results support the findings.

Critical weaknesses
  • The paper does not follow ICLR guidelines of formatting. Thus, it should be subject to desk-rejection.

  • The writing is confusing. The paper would benefit from through rewriting.

  • L 209: “thus r > 1 means that the reward asymptotically exceeds this cost”. I am not sure what r is? Additionally, the choice of R and c is very ad-hoc.

  • Caption of Fig 3 mentions that dashed vertical lines mark maximizers in {0.25, 0.255, …1}, but the markings are in {0.8, 1}. In fact, 0.25 is not even on the x-axis.

  • The applicability of the setup is very niche. Theorem 4.2 works for extremely small values of alpha, and the success probability gains in the experiments is not huge. Still, this could benefit cases where extreme precision is required or the data collection is extremely costly. It would also be interesting to compare the expected stopping times in addition to the success probability. In general, it would be good to add a table of notations.

  • "Check that the initial Bellman continuation value is strictly positive for the specified reward and sampling cost"; though the authors have disclosed in the AI statement that they have used GenAI for writing and editing, this line seems like a todo left in the paper.

  • The problem looks a bit artificial, since the regulator doesn't have to bear any sampling cost, whereas in reality sampling for any regulator is costly.

Questions for authors

Can you please respond to the weaknesses, specially the last three issues?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.