acceptodds
Under review as a conference paper at ICLR 2027

Paying for Better Data in Prediction-Powered Testing

Abstract

Automated judgments offer a cheaper alternative to expert labels for evaluating large language models, but their accuracy may depend on a provider's unobserved effort. We study whether paying for better judgments can reduce testing costs. Our design combines random audits with bonuses for agreement with experts and chooses the contract and sample size together. We show that the robust likelihood factor maximizing expected log evidence equals a prediction-powered betting factor with jointly optimized predictions and bet size. The test controls false positives in finite samples under arbitrary reporting and attains the optimal leading sample size under the induced review model. With uncertain review costs and accuracy, we characterize contracts and sample sizes ensuring participation, high effort, and prescribed power throughout specified ranges. A finite-sample cost criterion compares candidate designs with direct gold acquisition. Simulations identify when incentives reduce certified costs, and comparisons with prediction-powered baselines clarify the role of inference efficiency. An LLMBar study using observed judgments and assumed costs illustrates how a weaker accuracy gain can favor direct gold acquisition.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Prediction-powered inference, hypothesis testing, data acquisition, principal–agent models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZAHYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The setting is a test comparing two LLMs when expert labels are expensive and an automated review service can expend costly, unobserved effort to sharpen its judgments—effort that a flat per-judgment payment does nothing to encourage. The paper jointly selects an audit-and-bonus contract and a sample size, showing that the robust likelihood factor against the closest null is an optimized prediction-powered betting factor. In favorable synthetic high-effort regimes, the selected designs reduce certified testing cost, whereas in the real-data study the observed benefit from higher effort is too small for the contract to be worthwhile and the selection criterion instead favors direct gold-label acquisition.

Critical strengths
  1. The paper introduces a novel and well-motivated view of evaluation quality as an incentive-design problem rather than a fixed data-collection problem.
  2. Proposition 4.1 and Appendix E.1 constructively show that the KL-projection factor equals the optimized prediction-powered betting factor, with the uniqueness/optimality argument built on strict concavity of log. This unifies two perspectives on the test and puts the proposed design on a rigorous footing.
Critical weaknesses
  1. The approach is technically complex and may be difficult for practitioners to calibrate or deploy without substantial statistical expertise.
  2. The real-data evidence does not demonstrate a setting in which the proposed payment mechanism is beneficial, limiting confidence in its practical impact.
  3. Tables 2, 8, and 11 show PPI/PPI++ achieving higher power at common budgets or lower cost, although the paper correctly notes that they lack the same finite-sample validity and can exceed nominal size. The practical tradeoff would be clearer with a systematic measurement of the magnitude and frequency of their size violations in the paper's regimes.
Questions for authors

In the synthetic and LLMBar regimes, how often and by how much do PPI/PPI++ violate nominal size, and is there a range where the proposed finite-sample method is cheaper than a size-corrected alternative?

1 comment

Thank you again for your thoughtful and constructive feedback. We hope that our responses and revisions have addressed your concerns. If you have any remaining questions or further comments, we would be very happy to respond and provide any clarification that would be helpful. We sincerely appreciate the time and care you have devoted to reviewing our paper.

Reviewer_yAovOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether paying an LLM judge for more careful work, through random expert audits plus an agreement bonus, can make a model-comparison test cheaper than buying expert labels. It pairs this contract with an audit-corrected e-value test that controls false positives no matter how the judge reports. In the experiments, paying only pays off in 4 of 10 synthetic settings, and the LLMBar replay, which uses assumed costs, recommends buying gold labels directly.

Critical strengths

It’s a nice framing: prediction quality becomes something you buy. Validity comes only from the audits, so a wrong guess about the provider costs money but never false positives. The paper is also careful, reproducible, and honest about its negative results.

Critical weaknesses
  1. The results look driven by a conservative test, not by the economics. In a quick check, computing power exactly instead of with the Chernoff bound roughly halves the sample size. The gold baseline also uses a tighter test than the proposed method. In the LLMBar replay, plain PPI already beats buying gold.
  2. The pieces feel stitched together. The core is classic double sampling (Neyman, 1938; Cochran), which isn’t cited. The only real link between incentives and statistics is the lower bound on the audit rate. Beyond that:
  • The main result, the log-optimal factor in Prop. 4.1, is never picked in practice.
  • The asymptotic theory doesn’t match the finite-sample decisions.
  • The incentive mechanism is never tested on real data.
Questions for authors
  1. How do the conclusions change if you use exact power and give the gold baseline the same kind of test?
  2. Beyond the audit-rate lower bound, what does the method add over classical double sampling?
Reviewer_oAHpOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper concerns the joint design of an incentive contract, sample size, and hypothesis test to evaluate the mean of Bernoulli random variables. This is applied to an automated review provider whose accuracy depends on some amount of effort chosen. A principle intends to choose certain parameters to incentivize high-effort reviewing and minimize error.

The contributions include a projection-betting identity which seems quite interesting, and a characterization in a particular uncertain setting. The authors also include empirics for their procedure.

Critical strengths
  • The formulation seems fairly clean, and in particular it is quite useful that the finite-sample Type I error guarantee holds unconditionally under arbitrary (even history dependent) reporting.
  • Even when the high-effort cost and error rate are unknown, the paper still finds a useful salvage.
Critical weaknesses
  1. The assumptions of symmetric errors and a known low-effort error rate seem fairly substantial. Under asymmetric errors, the provider's expected agreement probability is no longer independent of the unknown win rate μ\mu, which complicates the prevalence-free incentive condition. Moreover, in the LLMBar experiment, the low-effort error is not known a priori and must be estimated alongside the high-effort error (with both shifting noticeably between calibration and evaluation and exhibiting asymmetry); while this departure is briefly noted in Appendix H.3, it should be highlighted in the main text as it directly affects whether the theoretical guarantees apply in practice.
  2. There seems to be a significant mismatch between the Markov inequality (and Chernoff power bound) used to certify the proposed test and the exact randomized binomial test used for the gold-only baseline. Because the e-value test relies on conservative concentration bounds (yielding empirical Type I error around 0.002–0.0080.002\text{--}0.008 at nominal α=0.05\alpha = 0.05 and sample sizes 3.1×–3.7×3.1\times\text{--}3.7\times the asymptotic leading term), much of the apparent cost disadvantage against direct gold acquisition in finite samples (Table 7) stems from bound looseness rather than the underlying economic mechanism.
Questions for authors
  1. It seems that the following reference is quite relevant but not cited: https://arxiv.org/abs/2607.08347. The following references also seem quite relevant: https://arxiv.org/pdf/2406.02873, https://arxiv.org/abs/2603.27414
  2. Since the observation OiO_i takes values in a 6-cell multinomial and the null reduces to a 2-parameter family (x0,x1)∈(0,1)2(x_0, x_1) \in (0,1)^2, could you calibrate the rejection threshold and power directly from the finite-sample multinomial distribution (or a randomized threshold) rather than Markov + Chernoff bounds? How much of the gap against the exact gold-only binomial test in Table 7 is just concentration-inequality slack?
  3. Asymmetric Errors and Estimated ηL\eta_L: Can the incentive condition (Eq. 19) and four-corner power bound (Theorem 5.1) be extended when false-positive and false-negative rates differ (ηe,0≠ηe,1\eta_{e,0} \neq \eta_{e,1}, as in LLMBar) and when ηL\eta_L is also subject to estimation uncertainty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.