- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether language models can verify solutions to problems they rarely solve themselves. It characterizes verifier behavior using informedness (Youden’s J) and leniency (L), distinguishing discrimination between correct and incorrect answers from the tendency to accept. Across six models and seven benchmarks, the authors find positive informedness on hard mathematics and coding problems, but no statistically significant departure from zero on hard knowledge questions. The paper also studies committees formed by repeated judgments from the same model. It develops mean-only and Beta approximations to predict committee performance and select voting thresholds. Both approaches improve informedness over majority voting on average, while the Beta approximation more accurately predicts committee operating curves.
Critical strengths
-
The paper investigates a practically important question: whether language models can assess solutions to problems they rarely solve themselves. Its evaluation across six models and seven benchmarks reveals task-dependent differences in verification ability, providing useful evidence about when self-verification may support scalable oversight.
-
The use of informedness and leniency provides an interpretable characterization of assessor behavior. By distinguishing discrimination between correct and incorrect solutions from the tendency to accept, this framework helps avoid misleading conclusions based on accuracy alone.
Critical weaknesses
-
Limited methodological novelty. Informedness (Youden’s J) is an established metric, and its combination with leniency is an easy reparameterization of TPR and FPR. The committee analysis also builds on standard binomial aggregation and moment matching. While these tools provide a useful perspective, the paper does not sufficiently establish what methodological contribution goes beyond combining existing concepts.
-
Unclear motivation and organization. The introduction is too brief to clearly explain the motivation, the limitations of existing work, and how the proposed approach addresses them. Although the introduction identifies limitations of prior studies, it does not sufficiently explain how the individual components of the proposed framework address these limitations or connect to one another. Besides, Section 3 introduces several concepts, including utility, the generation–verification gap, and randomized verdict flipping, before making their roles in the subsequent analysis clear. This organization makes the central argument difficult to follow and obscures the distinction between supporting analysis and the main methodological contribution. Some abbreviations are also used without being spelled out at first mention, further reducing readability.
-
Limited comparison with existing methods. The related work discusses several verifier aggregation approaches, but the experiments mainly compare against simple baselines such as a single assessor and majority voting. Comparisons with relevant existing methods under matched computational budgets would help establish the practical value of the proposed approach.
Questions for authors
-
The first limitation concerns benchmark-level aggregation mixing problems of different difficulty. Would stratifying questions by the model’s solve rate be sufficient to address this limitation?
-
Do the authors provide any goodness-of-fit analysis showing how well the Beta distributions capture the class-conditional distributions of acceptance propensities, beyond the reported prediction errors for committee-level informedness and leniency?
-
Can the authors provide theoretical guarantees on how Beta approximation and estimation errors affect committee performance predictions and threshold selection?
-
The fitting and evaluation stages use different verdicts on the same proposals. Do the prediction accuracy and threshold-selection gains persist when calibration and evaluation use disjoint questions?
Author Response to Reviewer u2kn
We thank the reviewer for their thoughtful comments. We appreciate their recognition of the practical importance of studying self-verification beyond a model's own solving capability, and of the value of separating informedness from leniency to characterize assessor behavior. We have revised the analysis and presentation in response to their concerns, which we address below.
Responses to weaknesses:
Methodological novelty. We agree that Youden's J, the informedness-leniency parameterization, and classical committee theory build on established statistical concepts. Our contribution lies in using these tools to study self-verification conditioned on a model's own per-question solving ability, characterize the resulting task-family differences, and predict and select committee operating points.
Motivation and organization. We have reorganized the paper to distinguish the main empirical contributions from supporting theory. In particular, we moved the utility interpretation and generation-verification gap analysis to Appendix E. The main Methods section now focuses on experimental procedures, committee prediction, and threshold selection.
Comparison with existing methods. We study homogeneous committees formed from repeated binary verdicts of one model. Within deterministic, monotone aggregation rules that depend only on the acceptance count, the problem reduces to selecting a threshold k. We compare majority voting, mean-only and Beta-selected thresholds, and the best observed threshold as an optimistic reference (Table 2). These committee methods use the same nine verdicts per proposal at evaluation time, although calibration requires additional measurements. We also compare against randomized verdict flipping at matched leniency. Methods using heterogeneous verifiers or richer assessment signals are outside this setting, and we do not claim superiority over them.
Responses to questions
Would stratifying by solve rate address the first limitation? Yes, it addresses the problem of benchmark-level averages mixing questions of different difficulty. We estimate per-question solve rates from 100 proposals and examine verification across difficulty bins (Figure 1) and cutoffs (Table 1). We also use J to measure discrimination independently of class prevalence. Stratification does not, however, eliminate differences in proposal format or error characteristics across tasks.
Beta goodness-of-fit analysis. We have added a diagnostic comparing predicted and observed acceptance-count distributions in held-out nine-member committees (Section 4.2 and Appendix D.1). The Beta approximation achieves KL divergences of 0.028 for correct and 0.019 for unsuccessful proposals, compared with 1.428 and 1.709 for mean-only. Beta performs better in all 40 valid model-benchmark comparisons for each class. We also simulate verdicts under the fitted Beta model to estimate the divergence expected from finite sampling. The remaining discrepancy, particularly for correct proposals, indicates that the approximation is not exact. This diagnostic evaluates observable acceptance-count distributions rather than latent acceptance propensities directly.
Are there theoretical guarantees? We do not establish guarantees for the fitted Beta approximation. Matching the first two moments does not uniquely determine the higher moments required for committee prediction, so accuracy of the Beta approximation requires additional assumptions. We instead evaluate it empirically using held-out verdicts, distributional goodness-of-fit, and threshold-selection performance, including on disjoint questions. Establishing guarantees under specific distributional assumptions remains an open direction.
Do the gains persist on disjoint questions? Yes. As we also outline in our response to reviewer xqMj, we now include five-fold question-disjoint cross-validation (Section 4.2 and Appendix B.5). The mean-only and Beta-selected committees retain gains of ΔJ = 0.046 [0.036, 0.058] and 0.043 [0.030, 0.057], respectively, compared with 0.011 [-0.008, 0.028] for majority voting. On held-out questions, Beta also predicts the operating curves more accurately than mean-only, with RMSEs of 0.121 versus 0.192 in J and 0.077 versus 0.199 in L. Thus, the gains from threshold selection largely persist, although curve prediction becomes less accurate than in the same-proposal evaluation. Generalization across different question distributions remains untested and is an interesting direction for future work.