- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether a small number of repeated responses can help decide if further sampling is worthwhile. It connects paired correctness observations to moment constraints on example success probabilities, then derives the best and worst voting accuracies consistent with those constraints. The main results include explicit bounds for three votes, a numerical method for other finite budgets, and bounds for the limiting accuracy. The experiments examine voting gains, changes in model rankings, and forecasts obtained from moment matching.
Critical strengths
-
The bounds make a useful distinction between estimating the moments accurately and identifying the voting curve. Even with exact moments, different populations can have different voting gains. The attaining distributions show that this ambiguity is unavoidable with the available information, which gives the bounds a clear interpretation.
-
The authors are careful about what the experiments establish. Q04 and N05 are fixed before evaluation on separate data, and the appendix explicitly distinguishes checks that reuse responses from independent coverage validation. The matched direct prompting control for N05 is also reported, even though it outperforms the randomized protocol after voting.
Critical weaknesses
-
The experiments do not establish when collecting two responses on more examples is preferable to collecting more responses on fewer examples. Q04 and N05 demonstrate useful certificates, but the GSM8K bounds cannot resolve the observed gain at nine calls. Some uncertainty comes from knowing only two moments and would remain with more validation examples. A comparison under the same total call budget is needed to assess the advantage over directly estimating voting gains.
-
In Table 10, the moments are estimated from the first two responses, while empirical voting accuracies use all five. LDGP’s small errors therefore do not yet establish prediction accuracy on separate response records. The confidence intervals also rely on the Wald approximation, whose coverage is not assessed by the checks in Appendix B.5 or Table 7, as the authors acknowledge. Given the 0.5 percentage point decision threshold, separate checks of forecasting accuracy and interval coverage would help establish whether these conclusions are reliable.
Questions for authors
-
Under the same total validation call budget, how does collecting two responses per example compare with collecting five responses on fewer examples? The latter can estimate both three vote and five vote gains using the subset estimator in Appendix C. How often does each design correctly resolve the 0.5 percentage point threshold?
-
For GSM8K, could you report gain intervals at three, five, and nine calls, both with the estimated moments fixed and after Wald projection? This would distinguish uncertainty from moment estimation from the ambiguity that remains even when the moments are known.
-
Could you check Wald region coverage through multinomial simulations at the sample sizes and pair probabilities of Q04, N05, and N04, including nearby probabilities? This requires no additional LLM calls and would help assess calibration around the 0.5 percentage point threshold.
-
Could Table 10’s three vote forecasts be evaluated by fitting the moments on two responses and measuring majority accuracy on the remaining three? The existing records allow this check. Please include uncertainty in the errors, since using fewer evaluation responses will introduce additional noise.
4 comments
W2, Q3
Simulated Wald coverage is within about 0.3 points of nominal for Q04/N05 and below nominal for N04; an exact binomial check supports the N04 decision. Simulations use 20,000 multinomial replications/scenario, re-estimating covariance each time; N=6144 for Q04/N05 and 8192 for N04. Nearby scenarios multiply discordance by 0.8/1.2 and rescale the concordant cells:
| Setting | Nominal | 0.8x | Original | 1.2x |
|---|---|---|---|---|
| Q04 | 97.50% | 97.38% | 97.56% | 97.70% |
| N05 | 97.50% | 97.20% | 97.47% | 97.51% |
| N04 | 95.00% | 94.27% | 94.52% | 94.72% |
| N04 | 99.72% | 99.42% | 99.54% | 99.54% |
Q04/N05's 97.5% marginal levels give nominal 95% joint coverage; 99.72% is Bonferroni across 18 settings. MC SEs are 0.03–0.16 points.
For N04, 51/8192 disagreements give a one-sided Clopper–Pearson upper limit 0.0090 at level 1−0.05/18. The paper’s bounds give three-/five-vote gain upper limits 0.452/0.465 points, below 0.5, assuming iid item pairs and conditional-iid voting. The revised version adds the coverage simulation and the exact check (Appendix I.4).
Q4
On separate responses, three-vote LDGP forecasts have mean absolute error 0.288 points. Table 10 uses shared responses, so we fit LDGP on calls 1–2 and evaluate majority on calls 3–5; SEs account for item-level covariance through the joint influence function. The revised version adds this evaluation as Table 15, with each setting's error (positive for overprediction) and its 95% delta-method interval.
LDGP is a point forecast: it selects one latent distribution compatible with the moments, whereas the sharp interval covers every compatible population, so its accuracy is reported apart from Q3's coverage.
We hope these responses address your concerns, and we would be grateful if you would consider raising your score.
Thank you for the explanations and the effort you put into the rebuttal. Your responses helped clarify my understanding of the paper, and I have increased my score after considering the additional explanations.
Thank you for the careful reading of our responses and for raising your score. We are glad the explanations clarified the paper, and we would be happy to answer any further questions.
Thank you for your careful review. We appreciate your constructive comments.
We have uploaded a revised version: changed text is in blue, new appendix sections have blue titles, and red margin notes name the review point each change addresses.
W1, Q1
A validation design's cost is set by the budgets it must cover. To compare budgets up to K, direct repeats take K calls on each of N examples and identify accuracy at budgets up to K; the infinite-vote endpoint would require calls per example to grow along with N. Two labeled calls per example require only N to grow: 2N calls estimate the exact envelopes at every odd budget and at the infinite-vote endpoint (Theorems 4.1 and 4.3). With our N=8192, budgets up to nine calls take 9N=73,728 calls by direct repeats and 2N=16,384 by pairs.
We ran your equal-cost comparison at prespecified budgets. On the existing records, 8190 calls give 4095 pairs (P) or 1638 five-call examples (D), with 1000 row resamples per condition. P samples a uniform pair per record and projects a nominal 95% bivariate Wald region onto both budgets; D estimates subset gains with two 97.5% paired-normal intervals (95% jointly). A correct resolution places the entire interval on the same side of the 0.5-point threshold as the empirical gain.
P resolves N04, a gain below the target, in 87.2%/83.7% of replications at three/five votes and Q05 in 24.3%/18.8%; D resolves these in 100%/100% and 100%/99.8%, and the other 16 settings, whose envelopes contain the target, in 3.4–100%; at a prespecified budget D resolves at least as often as P throughout. The maximum wrong-resolution rate is 0.7% (MC SE below 1.6 points), conditional on the records.
Appendix I.1 gives the first-order criterion. With m_P the margin by which the envelope clears the target, m_D≥m_P that of the gain, s_P, s_D the per-example standard deviations and c_P, c_D the critical values, pairs need fewer calls when
2(cPsP/mP)2<K(cDsD/mD)2,
that is, when the envelope is tight relative to the gain or K is large, since the left side does not grow with K. When the envelope contains the target, only direct repeats resolve the decision.
On the separate Q04/N05 triple records, 3072 calls give 1536 pairs or 1024 triples; over 1000 resamples, pair/triple correct-resolution rates are Q04 8.9%/92.8% and N05 66.3%/100% (MC SEs at most 1.49 points). The designs thus work in sequence: pairs give the envelope across budgets for 2N calls, and direct repeats verify the budget chosen from it. The revised version adds this comparison and the cost across budgets (Section 6.1; Appendix I.1).
Q2
The paper determines exactly when the two moments settle the sign and what settles it otherwise. By the finite-support result (Theorem 4.1), the envelope at every budget is exact for every moment pair, attained by laws with at most three support points. Theorem 4.2 turns this into an explicit criterion: for mu>1/2, three-vote accuracy exceeds mu for every compatible population exactly when rho<1−1/(2mu). Q04 and N05 meet it, so their lower certificates are positive.
At the GSM8K estimates (mu=91.23%) the criterion fails, so the exact envelopes span both signs at three, five and nine calls, whatever the precision of mu and nu. The sign is then fixed by higher moments: three-vote accuracy depends on the latent law only through nu and the third moment (Section 4.2), and R labeled calls identify accuracy at every odd budget up to R; beyond R the finite-support envelopes extend with at most R+1 support points (Section 4, after Theorem 4.1). The nine-call record supplies these moments and shows strict-majority gains of 0.42, 0.61 and 0.85 points at three, five and nine calls (Table 6, Appendix D.2). The revised version reports the exact envelopes and their Wald projections at three, five and nine calls (Table 13); they bound strict-majority correctness, whereas the GSM8K evaluation also reports plurality.
Our reply to W2, Q3 and Q4 follows in a second comment.