acceptodds
Under review as a conference paper at ICLR 2027

What a Reported Pooled AUC Identifies When Scores Are Unavailable

Abstract

A pooled AUC over a stratified evaluation is often interpreted as evidence that a score discriminates well within strata. In general, however, the pooled AUC does not determine within-stratum discrimination. When per-example scores are unavailable, a reader may observe only the reported AUC, the outcome labels, and the stratum memberships. We study exactly what this information identifies, separating two distinct questions. First, what reference value should the reported AUC be compared against? We define its conditional permutation mean: the expected pooled AUC when labels are permuted within strata while scores remain fixed. This is not a new metric, but a reference value induced by the unreleased scores themselves, and it need not equal 0.5. From annotations alone, we derive sharp extrema for this quantity, attained by rankings that place strata in contiguous blocks. Conditioning additionally on an exact reported AUC yields tighter sharp bounds together with an explicit attaining construction. Second, what does the reported pooled AUC imply about within-stratum discrimination? We characterise exactly when it is compatible with within-stratum AUCs of 0 or 1. On PRM800K, the reported AUC substantially narrows the annotation-only range and can exclude 0.5 as the conditional permutation mean. Yet none of the seven AUCs we recompute from the released data rules out a within-problem AUC below 0.5. On ProcessBench, interpreting the published figure as a standard AUC, scores matching 0.765 at the reported precision attain within-solution AUCs of both 0 and 1. These results separate what a pooled AUC identifies from what it leaves undetermined and specify the per-stratum statistics sufficient to identify each quantity.

Primary area:
datasets and benchmarks
Keywords:
AUC, partial identification, evaluation methodology, stratified evaluation, conditional permutation mean, benchmark reporting, process reward models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bwYpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies, in two-class soft classification, the relationship between the AUC for a complete test set (called pooled AUC) and the AUC for a subset of it (called within-stratum AUC). By AUC, one should understand the area under the ROC (Receiver Operating Chatracteristic) curve obtained by thresholding the score predicted by the classifier. The pooled AUC is reported; the within-stratum AUCs are unknown. The relationship between them is studied through two questions. (1) What reference value should the pooled AUC be compared against? (2) What does the pooled AUC imply about within-stratum AUCs? The dataset PRM800K (« 800,000 step-level correctness labels on LLM solutions to MATH problems ») is taken as usecase.

Critical strengths

It is often necessary to obtain aggregate values for performance scores when classifiers have been evaluated on different sets of samples. The pooling discussed in this paper is one possible approach. It is therefore important to understand its implications. Aside from stratified datasets, I can think of a few other situations where this occurs: (1) in cross-validation; (2) in computer vision, where classifications can be performed at the pixel level, grouped into images, which are themselves grouped into videos, which are themselves grouped into datasets; (3) in domain adaptation, when we are interested in the performance of classifiers evaluated across multiple domains.

  • Forman, G., & Scholz, M. (2010). Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement. ACM Sigkdd Explorations Newsletter, 12(1), 49-57.
  • Piérard, S., & Van Droogenbroeck, M. (2020). Summarizing the performances of a background subtraction algorithm measured on several videos. In ICIP, pp. 3234-3238. IEEE.
Critical weaknesses

Presentation

  • [W1] Reading this paper requires the reader to make assumptions.

    • (1) The word « classification » does not appear anywhere, even though the entire paper focuses specifically on soft classification in the very particular case of two classes. This should be stated explicitly from the very beginning.
    • (2) The authors must specify what the acronym AUC stands for. Readers in our community will naturally assume it refers to « Area Under Curve ». But the area under which curve? In two-class classification, there are several (ROC curve, PR curve, precision-recall-gain curve, etc.). By analyzing the introductory example, one can conclude that the authors are referring to the area under the ROC curve, but this requires some effort from the reader.
  • [W2] The notations are not consistent. For exemple, the global AUC is sometimes denoted by AreportedA_{reported} (as in l. 120) and sometimes as ApooledA_{pooled} (as in l. 131).

  • [W3] Missing references to the works in which the 7 classifiers (scoring rule) listed in table 1 have been introduced.

Informativeness

  • [W4] The introductory example (l. 42-48) illustrates in fact a well-known statistical paradox described by K. Pearson in 1899, G. U. Yule in 1903 and E. Simpson in 1951, here in the specific case of the AUC. It is therefore necessary to cite the literature on the Yule-Simpson effect. The authors are familiar with this work, as they cite Simpson in suppl. mat. on page 18. However, the authors chose to completely ignore these works in the paper.

  • [W5] From my perspective, in its current form, this paper offers little new insight to readers who are classification professionals. It is clear that a central trend in performance (whether that trend is obtained through averaging or pooling) cannot under any circumstance serve as a bound (i.e., as the worst- or best-case scenario). In classification, if we look at the smallest possible scale (i.e., that of the sample), we nearly always have a non-informative classification. The random variable representing the ground truth becomes nearly always deterministic, as does the one representing the prediction. Therefore, in the information theory perspective there is null mutual information between them: they are independent at that scale. However, anyone who has ever worked with classifiers knows that on datasets with multiple samples, it is possible to obtain non-zero mutual information between the ground truth and the prediction! It is therefore not surprising at all that a classifier can be non-informative on a partition but informative on the overall dataset. This implies that it is not surprising at all that a classifier can have an AUC = 0.5 on all subsets and an AUC ≠ 0.5 on the overall dataset.

  • [W6] Does that mean we need a point of comparison other than 0.5 for the overall AUC? No, unless the classifier has information about the per-sample stratum. If the classifier knows the stratum and the positive prior for that stratum, it can adapt to it and, overall, perform better than a non-informative (random) classifier. To the best of my knowledge, such situations are exceptional. Is it the case when one evaluates mathematical proofs by steps (with PRM800K)?

  • [W7] The practical value of the reported results is unclear. It is not specified if some code will be publicly available to compute the answers to Q1 and Q2. In the absence of publicly relesed code, the paper does not present any practical algorithm for these computations. Also, the computational complexity is not specified. The practical limits are not clear: is it possible to compute the two answers in a reasonable amount of time in all situations? Is it much less costly to do the proposed computations instead of performing a new evaluation of the classifier to obtain the missing per-stratum AUCs?

Soundness

  • [W8] Knowing how to calculate ν\nu is one thing; knowing how to use it is another. On lines 178 and 181, it is mentioned that this value is simply subtracted from the reported overall AUC. Such a correction for chance is clearly inappropriate as the result is not interpretable. If the overall AUC is 1, then we have a perfect classifier and want to retain the maximum score. Furthermore, if the overall AUC is 0, then we have a perfect classifier if we reverse the decisions, and we want to retain that information. There is a need to study a correction for chance applied to the overall AUC such that 0 → 0, nu → ½, and 1 → 1.

  • [W9] The paper fixates on the value 0.5. But what does it mean? The paper doesn’t say. Let’s assume the authors have in mind the case of non-informative classifiers (sometimes called random or no-skill classifiers). In the very specific case where the ROC curve is convex, an AUC of 0.5 implies that the curve is the ROC diagonal, which corresponds to a non-informative classifier (zero mutual information between the ground truth and the prediction), which is what can be obtained by chance when the axis of ROC (FPR and TPR) are probabilities. This highlights two fundamental problems. First, there is no assumption of convex curves anywhere in the paper. An AUC value of 0.5 therefore does not indicate that we have a non-informative classifier. Second, when there are few samples, the FPR and TPR values are proportions, not probabilities. Any point on the ROC space can be obtained with a classifier that behaves purely randomly. For these reasons, the value of 0.5 is meaningless as a point of comparison for the AUC.

Questions for authors

Questions

[Q1] Can you explain why, if one is interested by some given strata only, she or he would not evaluate the classifier on these strata but rather only consider an AUC computed on all strata ? Please give a concrete example.

[Q2] Can you please provide curves showing how the computation time evolves depending on the dataset size, for different numbers of strata? Can you also explain why the factorial (see line 238) is not a problem in practice ?

[Q3] Figure 1 shows a zonotope. A zonotope is typically a linear projection of some hyper-cube. The linearity of both axes comes from the linearity of ApooledA_{pooled} and ν\nu with respect to RiR_i (see eq. 3). But the possible vectors for RiR_i do not cover a hypercube. So can you explain why we have a zonotope in Figure 1?

[Q4] Can you explain in simple words why the minimum ν\nu value is not achievable when Apooled=0A_{pooled}=0 and why the maximum ν\nu value is not achievable when Apooled=1A_{pooled}=1?

Suggestion

Is the introductory example (l42-48) well chosen? It is a very specific case. If we swap the labels in the two strata, nu remains constant. This example therefore does not help the reader to understand why ν\nu can fall within a wide range.

2 comments

Thank you for the detailed review. Changes are in the revised PDF.

W1-W3. The first sentence now states the setting (two-class soft classification) and that AUC is the area under the ROC curve. Sec. 5.1 cites the source of each of the seven rules (Lightman et al., 2024; Wang et al., 2024; the last-step, mean and first-step rules are our aggregations). The pooled AUC is now A_pooled throughout.

W4. Agreed. The introduction now names the example as the Yule-Simpson effect (Pearson et al., 1899; Yule, 1903; Simpson, 1951), and Related Work cites the two suggested papers.

W5. We agree this is not surprising, and do not claim it. We also do not use a pooled figure as a bound. We ask how far the printed figure and per-stratum counts constrain ν and the within-stratum AUC, and the answer differs by design: on PRM800K, 0.941 confines ν to [0.562, 0.908]; on ProcessBench, 0.765 leaves ν in [0.434, 0.901] and allows within-solution AUC 0 and 1.

W6. Yes, and it is typical: an LLM verifier reads the problem, so it has the stratum information by construction. Measured ν is 0.607 to 0.797 on PRM800K and 0.636 to 0.735 on Math-Shepherd. Within-problem discrimination is also real on PRM800K (within-problem AUC 0.884 for the ORM); the point is that the printed 0.941 alone does not establish it.

W7, Q2. The algorithm (Sec. 4, Appendix A) evaluates a dual by one O(n log n) sort over a fixed number of search iterations, so the computation is O(n log n). New Table 4 gives run times (as a table, not curves): one endpoint takes 0.1 to 0.3 s at n = 10^5 and 1.2 to 2.6 s at n = 8 x 10^5, about twice as long with 10^4 strata as with 10. The factorial appears only in exhaustive checks on at most eight items. The code is in the supplementary archive and will be released publicly with the paper. A new evaluation needs the scores, which a reader of a published figure does not have.

W8. We do not propose A_pooled minus ν, or the suggested rescaling, as a corrected measure; Sec. 3 now says so. Neither recovers within-stratum discrimination: in the eight-item design (Sec. 4), A_pooled = 0.5625 and ν = 0.5, so any such rescaling gives more than 1/2, yet A_within = 1/3.

W9. Sec. 3 now states what 0.5 means here: the permutation mean of an AUC inside one stratum, and the value a pooled AUC is commonly read against. We assume no convex ROC curve and never equate 0.5 with a non-informative classifier. ν is an exact finite-sample permutation expectation, so nothing treats FPR and TPR as probabilities.

Q1. Lee et al. report 0.765 on ProcessBench MATH. Their step scorer is used to find the first wrong step inside a solution, so a reader needs the within-solution AUC. But their code pools all steps into one AUC, and the scores are not released, so the reader cannot evaluate per solution.

Q3. The hull of the points in Figure 1 is a zonotope because the permutohedron, whose vertices are the rank vectors, is itself a zonotope (the Minkowski sum of the segments [e_i, e_j]), and a linear image of a zonotope is a zonotope.

Q4. The largest ν needs strata in contiguous blocks ordered by share of positives (Proposition 1). A_pooled = 1 needs every positive above every negative. With mixed strata the two conflict; by symmetry the same holds for the minimum at 0. In the eight-item design, ν reaches its maximum 0.75 at every pooled AUC from 9/16 to 15/16, but at A_pooled = 1 its maximum is 0.71875 (checked over all 8! orderings).

Suggestion. Agreed. The introduction now adds the mirror case: with the same labels, scoring the hard problem above the easy one gives ν = 0.10 instead of 0.90.

Ethics concern 1 (doubtful results). The sentence that no check certifies how labels and strata were read discloses a limitation; it does not indicate an error. We have now checked our parsing against published counts. ProcessBench MATH: 594 erroneous and 406 correct solutions, matching the benchmark paper's Table 2. PRM800K: 500 problems with at most 1,860 samples each, as stated in the release's README. The README gives no total, so our total of 815,632 solutions is checked only by rebuilding it from the release, which reproduces it exactly. The scripts are in the revised supplementary archive.

Ethics concern 2 (AI assistance). Our wording caused a misunderstanding. The AI Use Statement already disclosed that the scripts, including verify_artifact.py, were written with AI assistance; "independent of that assistance" was meant to say that the exhaustive enumerations were implemented separately from the solver, not that a human wrote them. The proofs were checked by one of the authors; the scripts were checked by running them and comparing their outputs with the released data. The revised AI Use Statement says this explicitly.

I keep my opinion that the paper offers little new insight to readers who are classification professionals. Moreover, it remains unclear what one can do with the value of ν\nu once it has been computed, and how the interpretation we give to the AUC should be corrected (for chance) based on this value. Also, the paper still requires substantial reading efforts in places. Therefore, I maintain my rating.

Reviewer_auZzOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates what can be inferred from a reported pooled AUC when individual prediction scores are unavailable. The authors derive bounds on the expected pooled AUC under within-stratum label permutations and characterize which within-stratum AUC values are compatible with the reported statistic. The framework is evaluated using several public benchmarks, including PRM800K and ProcessBench.

Critical strengths

None

Critical weaknesses

W1. Poor presentation. The manuscript is unnecessarily complicated and difficult to follow. Extensive theoretical and implementation details obscure the relatively simple central message.

W2. Limited significance. The distinction between pooled and within-group AUC is well understood. It is unclear whether recovering bounds from an aggregate AUC without the underlying scores addresses an important practical problem, particularly when reporting within-group AUC directly would be a simpler solution.

W3. Limited novelty. The main observation follows from standard AUC decomposition and aggregation bias. Although the paper derives additional bounds, their significance beyond this established observation is unclear.

Questions for authors

Q1. What important practical use case motivates recovering bounds from a published pooled AUC rather than directly evaluating or reporting within-group AUC?

Q2. What are the main contributions beyond existing work on AUC decomposition and aggregation bias?

Q3. Could the authors substantially simplify the presentation and clarify the main contributions?

3 comments

Thank you for the review.

W2, Q1 (use case). Reporting within-group AUC, or per-stratum counts, is the better practice, and it is our recommendation (Sec. 6). The use case is figures already published without scores. Lee et al. print 0.765 for step-level scoring on ProcessBench MATH and release no scores, so no within-solution statistic can be computed. From the printed figure and the labels, ν lies in [0.434, 0.901] and within-solution AUC 0 and 1 both reproduce 0.765.

We do not claim the problem is frequent: of 506 papers coded, 34 were flagged, 6 confirmed exposed after full reading, and one allows the bounds to be computed. We also do not claim this settles how practical the problem is. That number is small partly because of what is released: none of the 6 releases per-example scores, and for most the per-stratum counts are not public either. Our recommendation is to print the counts beside a pooled figure so that the question can be answered on more papers.

W3, Q2 (contributions). We do not claim the pooled versus within-group distinction. We contribute (i) sharp bounds on the reference value ν from the printed AUC with an attaining ranking, (ii) the exact condition for within-stratum AUC 0 or 1, (iii) two audits (PRM800K, ProcessBench), and (iv) the per-stratum statistics that make an audit possible without scores.

W1, Q3 (presentation). The revised PDF restructures the introduction around the central message (the printed figure alone does not show whether a score ranks well inside a group) and a summary table, and moves tie, rounding and computation details to Appendix A.

Thank you for the rebuttal. I still find the paper extremely hard to follow, starting from the very first sentence of the abstract. Could the authors try to explain their ideas in a more human-readable way? What is the motivation? What are the main challenges? What is the actual contribution?

Thank you for reading the rebuttal and for saying this directly. In one sentence, the paper asks what a printed pooled AUC tells a reader when the scores behind it are not released.

Motivation. Some papers on step-level verifiers report one AUC computed over all steps of all solutions pooled together. Readers take a high value to mean the verifier can tell correct steps from wrong ones inside a solution, which is what it is used for. A pooled AUC can also be high because the verifier gives higher scores to easy problems, whose steps are mostly correct, than to hard ones, and that says nothing about ranking inside a solution. This is the Yule-Simpson effect. When the per-step scores are released, prior work handles it: compute the within-solution AUC and check. Our case is the one where the scores are missing and the reader has only the printed number, the step labels, and which steps belong to which solution.

Main challenge. Without the scores the usual tools break down. Splitting the AUC into within-solution and between-solution parts needs the scores. Comparing the number with 0.5 can also mislead, because the pooled AUC a score would get with labels shuffled inside each solution depends on how the unreleased scores order the solutions, and it can be far from 0.5. So the question is what can be proven from the printed number alone.

Contribution.

  1. We take as reference value ν the pooled AUC we would expect if labels were shuffled within each solution while the scores stay fixed, and use it only as a point of comparison for the printed number. From the printed number and the labels we compute the narrowest possible range for ν, and show each end is reached by an actual ranking.
  2. We give an exact test of whether a printed number is compatible with a verifier that ranks steps perfectly, or completely backwards, inside every solution.
  3. We apply both to published results. For ProcessBench MATH, the printed 0.765 is compatible with both, so a reader cannot tell from it whether the verifier ranks steps inside a solution at all. For PRM800K, the printed 0.941 rules out 0.5 as the reference value (ν in [0.562, 0.908]), yet a score that reproduces 0.941 can have a within-problem AUC of 0.175.
  4. We state which per-solution counts a paper should print beside a pooled AUC so that a reader can answer this without the scores.

We agree the abstract still opens with technical terms. The revision rewrote the introduction along these lines and left the abstract as it was. We will rewrite the abstract in the same plain form, starting from the motivation above.

Reviewer_aHRjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper investigates what can be inferred from a reported pooled AUC when individual prediction scores are unavailable. It studies two questions: identification of the conditional permutation mean and the compatibility of the reported AUC with different levels of within-stratum discrimination. The work develops sharp bounds and explicit ranking constructions, with applications to PRM800K and ProcessBench.

Critical strengths

S1. The paper addresses a relevant evaluation problem: a high pooled AUC does not necessarily imply strong within-stratum discrimination. Its partial-identification analysis attempts to characterize what can be inferred when individual prediction scores are unavailable.

Critical weaknesses

W1. Limited novelty and significance. The distinction between pooled AUC and within-stratum discrimination is already established in prior work. Although the paper derives sharp identification bounds, the extent to which these results provide substantially new practical insights remains unclear.

W2. Difficulty in assessing the technical contribution. Despite careful reading, I found the presentation unnecessarily complicated. The central contribution is difficult to distinguish from existing AUC decomposition and conditional evaluation results, making it challenging to assess the significance of the technical developments.

Questions for authors

Q1. What is the main technical advance over existing AUC decomposition and conditional evaluation methods, beyond the formulation of the partial-identification problem?

Q2. Could the authors provide a concise example demonstrating when the proposed sharp bounds lead to a practically meaningful conclusion that simpler existing analyses cannot provide?

2 comments

Thank you for the precise review.

W1. We agree that the pooled versus within-stratum distinction is established, and we do not claim it. Our question is what a printed pooled AUC identifies when the scores are unavailable. We limit the practical claim to the ProcessBench case below and to a reporting recommendation. The setting is not frequent: of 506 papers we coded, 6 were confirmed exposed and one of them allows the bounds to be computed from public information.

W2. In the revised PDF the introduction now opens with the ProcessBench case, states the two questions in plain words, and gives the answers in a summary table (Table 1); the tie, rounding and computation details moved to Appendix A.

Q1. The decomposition needs the scores. Without them, we give (i) a reference value, the conditional permutation mean ν, (ii) sharp bounds on ν from the printed AUC, where sharp means an explicit ranking reproducing the figure attains each endpoint (Theorem 1), and (iii) an exact condition for which printed values admit within-stratum AUC 0 or 1 (Proposition 2).

Q2. Lee et al. print a pooled AUC of 0.765 on ProcessBench MATH, and the scores are not released. From labels and 0.765 alone: ν lies in [0.434, 0.901], so 0.5 is not excluded; and score vectors matching 0.765 at its printed precision have within-solution AUC 0 and 1 (witness table, Sec. 5.2). A reader cannot tell from 0.765 whether steps are ranked correctly inside a solution. That second conclusion comes from Proposition 2, not from the sharp bounds. Sharpness itself gives a certificate that no narrower interval is possible. Its numerical gain over the simpler unconstrained bound is small on our designs: none for the six PRM800K rules printed at 0.897 and above, and [0.236, 0.910] becoming [0.294, 0.838] for the first-step rule. There is a narrow region where sharpness changes the verdict: on the PRM800K design, printed values from about 0.875 to 0.880 are excluded from 0.5 by the sharp bound but not by the simpler one (e.g. 0.878 gives [0.503, 0.908] against [0.498, 0.909]).

Thanks for the response. I maintain my original rating.

Comments on the paper

1 comment

We uploaded a revised PDF. Changes, all in response to the reviews:

  1. Introduction restructured: it now opens with the ProcessBench MATH case (a printed 0.765 is compatible with within-solution AUC 0 and 1), extends the two-problem example to show that ν can be 0.90 or 0.10 with the same labels, states the two questions in plain words, and gives the answers in a summary table (Table 1).
  2. The first sentence now states the setting (two-class soft classification) and defines AUC (area under the ROC curve, Mann-Whitney form), and the symbol for the pooled AUC is now the same throughout.
  3. Tie handling, rounding of printed figures and computation details moved from Section 4 to the start of Appendix A; the main text keeps a short pointer.
  4. Section 5.1 cites the source of each of the seven PRM800K scoring rules.
  5. The introductory example is now identified as the Yule-Simpson effect, with citations to Pearson et al. (1899), Yule (1903) and Simpson (1951); Related Work adds Forman and Scholz (2010) and Piérard and Van Droogenbroeck (2020).
  6. Section 3 now states what the value 0.5 means in the paper, and that the difference between the pooled AUC and the permutation mean is not proposed as a chance-corrected measure.
  7. Appendix A adds a run-time table.
  8. The supplementary archive now includes the scripts and results for the run-time measurements (one endpoint evaluation takes seconds at n = 8 x 10^5), for a check of our ProcessBench MATH parse against the counts published in the benchmark paper, and for the narrow region of printed values where only the sharp bound excludes 0.5.

New Table 1 (summary) and Table 4 (run times) shift the numbering: Tables 2-3 were Tables 1-2 as submitted, and Tables 5-17 were Tables 3-15. No result has changed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.