- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The setting is a test comparing two LLMs when expert labels are expensive and an automated review service can expend costly, unobserved effort to sharpen its judgments—effort that a flat per-judgment payment does nothing to encourage. The paper jointly selects an audit-and-bonus contract and a sample size, showing that the robust likelihood factor against the closest null is an optimized prediction-powered betting factor. In favorable synthetic high-effort regimes, the selected designs reduce certified testing cost, whereas in the real-data study the observed benefit from higher effort is too small for the contract to be worthwhile and the selection criterion instead favors direct gold-label acquisition.
Critical strengths
- The paper introduces a novel and well-motivated view of evaluation quality as an incentive-design problem rather than a fixed data-collection problem.
- Proposition 4.1 and Appendix E.1 constructively show that the KL-projection factor equals the optimized prediction-powered betting factor, with the uniqueness/optimality argument built on strict concavity of log. This unifies two perspectives on the test and puts the proposed design on a rigorous footing.
Critical weaknesses
- The approach is technically complex and may be difficult for practitioners to calibrate or deploy without substantial statistical expertise.
- The real-data evidence does not demonstrate a setting in which the proposed payment mechanism is beneficial, limiting confidence in its practical impact.
- Tables 2, 8, and 11 show PPI/PPI++ achieving higher power at common budgets or lower cost, although the paper correctly notes that they lack the same finite-sample validity and can exceed nominal size. The practical tradeoff would be clearer with a systematic measurement of the magnitude and frequency of their size violations in the paper's regimes.
Questions for authors
In the synthetic and LLMBar regimes, how often and by how much do PPI/PPI++ violate nominal size, and is there a range where the proposed finite-sample method is cheaper than a size-corrected alternative?
Thank you again for your thoughtful and constructive feedback. We hope that our responses and revisions have addressed your concerns. If you have any remaining questions or further comments, we would be very happy to respond and provide any clarification that would be helpful. We sincerely appreciate the time and care you have devoted to reviewing our paper.